2020-11-29T15:44:14.453 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-11-29T15:44:14.455 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --verbose --owner scheduled_mkogan@teuthology --block --name mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi --archive /home/teuthworker/archive/mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi/5665427 --lock -- /tmp/teuthology-worker.6gafeo7n.tmp 2020-11-29T15:44:14.484 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-29T15:44:14.553 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi/5665427 branch: wip-rgw-d3n-cache-upstreaming_i20 description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: mkogan@redhat.com first_in_suite: false job_id: '5665427' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-rgw-d3n-cache-upstreaming_i20 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 10684a45d9b8cdfc272d8721720e890aa44052c6 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 10684a45d9b8cdfc272d8721720e890aa44052c6 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-rgw-d3n-cache-upstreaming_i20 sha1: 10684a45d9b8cdfc272d8721720e890aa44052c6 owner: scheduled_mkogan@teuthology priority: 150 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 10684a45d9b8cdfc272d8721720e890aa44052c6 sleep_before_teardown: 0 suite: rgw suite_branch: wip-rgw-d3n-cache-upstreaming_i20 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-d3n-cache-upstreaming_i20/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 10684a45d9b8cdfc272d8721720e890aa44052c6 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2020-11-29_15:41:09 tube: smithi user: mkogan verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.25223 2020-11-29T15:44:14.554 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-d3n-cache-upstreaming_i20/qa; will attempt to use it 2020-11-29T15:44:14.554 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-d3n-cache-upstreaming_i20/qa/tasks 2020-11-29T15:44:14.555 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-29T15:44:14.556 INFO:teuthology.task.internal:Checking packages... 2020-11-29T15:44:14.578 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '10684a45d9b8cdfc272d8721720e890aa44052c6' 2020-11-29T15:44:14.579 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-29T15:44:14.579 INFO:teuthology.packaging:ref: None 2020-11-29T15:44:14.579 INFO:teuthology.packaging:tag: None 2020-11-29T15:44:14.580 INFO:teuthology.packaging:branch: wip-rgw-d3n-cache-upstreaming_i20 2020-11-29T15:44:14.580 INFO:teuthology.packaging:sha1: 10684a45d9b8cdfc272d8721720e890aa44052c6 2020-11-29T15:44:14.580 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-rgw-d3n-cache-upstreaming_i20 2020-11-29T15:44:14.742 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7606-g10684a45-1bionic 2020-11-29T15:44:14.742 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-29T15:44:14.752 INFO:teuthology.task.internal:no buildpackages task found 2020-11-29T15:44:14.752 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-29T15:44:14.767 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-29T15:44:14.768 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-29T15:44:14.824 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-29T15:44:24.841 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-29T15:44:34.859 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-29T15:44:44.877 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-29T15:44:54.894 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-29T15:45:04.910 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-29T15:45:14.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-29T15:45:24.943 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_mkogan@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi/5665427'} 2020-11-29T15:45:24.964 DEBUG:teuthology.lock.ops:locked ubuntu@smithi168.front.sepia.ceph.com, ubuntu@smithi184.front.sepia.ceph.com 2020-11-29T15:45:25.012 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3842355a60>, signals=[15]) 2020-11-29T15:45:25.257 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of ubuntu 18.04 2020-11-29T15:45:25.278 INFO:teuthology.provision.fog.smithi168:Scheduling deploy of ubuntu 18.04 2020-11-29T15:45:25.741 INFO:teuthology.orchestra.console:Power off smithi184 2020-11-29T15:45:25.741 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-29T15:45:25.762 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:25.815 INFO:teuthology.orchestra.console:Power off smithi168 2020-11-29T15:45:25.816 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-29T15:45:25.836 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:25.900 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2020-11-29T15:45:26.001 INFO:teuthology.orchestra.console:Power on smithi184 2020-11-29T15:45:26.001 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-29T15:45:26.022 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:29.856 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:30.042 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:33.968 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:34.152 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:34.363 INFO:teuthology.orchestra.console:Power on for smithi184 completed 2020-11-29T15:45:34.464 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish 2020-11-29T15:45:38.078 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:38.287 INFO:teuthology.orchestra.console:Power off for smithi168 completed 2020-11-29T15:45:38.388 INFO:teuthology.orchestra.console:Power on smithi168 2020-11-29T15:45:38.388 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-29T15:45:38.410 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:42.434 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:46.546 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi168.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-29T15:45:46.756 INFO:teuthology.orchestra.console:Power on for smithi168 completed 2020-11-29T15:45:46.857 INFO:teuthology.provision.fog.smithi168:Waiting for deploy to finish 2020-11-29T15:48:23.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:48:32.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:48:41.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:48:49.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:48:50.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:48:58.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:48:59.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:07.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:08.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:16.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:17.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:25.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:26.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:34.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:43.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:50.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:52.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:49:59.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:02.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:06.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:06.057 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:50:12.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:12.075 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:50:18.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:18.087 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:50:24.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:25.127 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:50:31.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:31.293 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-29T15:50:31.295 INFO:teuthology.orchestra.run.smithi184:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-29T15:50:31.743 INFO:teuthology.provision.fog.smithi184:Node is ready 2020-11-29T15:50:31.743 INFO:teuthology.orchestra.run.smithi184:> hostname 2020-11-29T15:50:31.789 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184.front.sepia.ceph.com 2020-11-29T15:50:31.789 INFO:teuthology.orchestra.run.smithi184:> grep smithi184.front.sepia.ceph.com /etc/hosts 2020-11-29T15:50:31.838 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.15.184 smithi184.front.sepia.ceph.com smithi184 2020-11-29T15:50:31.841 INFO:teuthology.orchestra.run.smithi184:> sudo hostname smithi184 2020-11-29T15:50:31.900 INFO:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/smithi184.front.sepia.ceph.com/smithi184/g' /etc/hosts 2020-11-29T15:50:31.959 INFO:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/172.21.15.184/172.21.15.184/g' /etc/hosts 2020-11-29T15:50:32.019 INFO:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/smithi184.front.sepia.ceph.com/smithi184/g' /etc/hostname 2020-11-29T15:50:32.073 INFO:teuthology.orchestra.run.smithi184:> sudo hostname smithi184 2020-11-29T15:50:32.129 INFO:teuthology.provision.fog.smithi184:Deploy complete! 2020-11-29T15:50:32.130 DEBUG:teuthology.parallel:result is None 2020-11-29T15:50:33.551 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:50:39.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:39.567 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:50:45.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:45.581 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:50:51.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:51.596 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:50:57.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:50:57.610 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-29T15:51:03.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:51:03.709 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-29T15:51:03.709 INFO:teuthology.orchestra.run.smithi168:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-29T15:51:04.183 INFO:teuthology.provision.fog.smithi168:Node is ready 2020-11-29T15:51:04.183 INFO:teuthology.orchestra.run.smithi168:> hostname 2020-11-29T15:51:04.232 INFO:teuthology.orchestra.run.smithi168.stdout:smithi168.front.sepia.ceph.com 2020-11-29T15:51:04.233 INFO:teuthology.orchestra.run.smithi168:> grep smithi168.front.sepia.ceph.com /etc/hosts 2020-11-29T15:51:04.281 INFO:teuthology.orchestra.run.smithi168.stdout:172.21.15.168 smithi168.front.sepia.ceph.com smithi168 2020-11-29T15:51:04.281 INFO:teuthology.orchestra.run.smithi168:> sudo hostname smithi168 2020-11-29T15:51:04.338 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/smithi168.front.sepia.ceph.com/smithi168/g' /etc/hosts 2020-11-29T15:51:04.394 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/172.21.15.168/172.21.15.168/g' /etc/hosts 2020-11-29T15:51:04.450 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i -e 's/smithi168.front.sepia.ceph.com/smithi168/g' /etc/hostname 2020-11-29T15:51:04.504 INFO:teuthology.orchestra.run.smithi168:> sudo hostname smithi168 2020-11-29T15:51:04.561 INFO:teuthology.provision.fog.smithi168:Deploy complete! 2020-11-29T15:51:04.561 DEBUG:teuthology.parallel:result is None 2020-11-29T15:51:04.697 INFO:teuthology.lock.ops:Checking smithi168.front.sepia.ceph.com 2020-11-29T15:51:04.698 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-29T15:51:04.713 INFO:teuthology.lock.ops:Checking smithi184.front.sepia.ceph.com 2020-11-29T15:51:04.713 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-29T15:51:04.727 INFO:teuthology.lock.ops:Updating [smithi168.front.sepia.ceph.com]: set os type and version on server 2020-11-29T15:51:04.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-29T15:51:04.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:51:04.802 INFO:teuthology.orchestra.run.smithi168:> true 2020-11-29T15:51:05.003 INFO:teuthology.orchestra.run.smithi168:> uname -m 2020-11-29T15:51:05.052 INFO:teuthology.orchestra.run.smithi168.stdout:x86_64 2020-11-29T15:51:05.052 INFO:teuthology.orchestra.run.smithi168:> cat /etc/os-release 2020-11-29T15:51:05.100 INFO:teuthology.orchestra.run.smithi168.stdout:NAME="Ubuntu" 2020-11-29T15:51:05.100 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-29T15:51:05.100 INFO:teuthology.orchestra.run.smithi168.stdout:ID=ubuntu 2020-11-29T15:51:05.101 INFO:teuthology.orchestra.run.smithi168.stdout:ID_LIKE=debian 2020-11-29T15:51:05.101 INFO:teuthology.orchestra.run.smithi168.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-29T15:51:05.101 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_ID="18.04" 2020-11-29T15:51:05.101 INFO:teuthology.orchestra.run.smithi168.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-29T15:51:05.101 INFO:teuthology.orchestra.run.smithi168.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-29T15:51:05.102 INFO:teuthology.orchestra.run.smithi168.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-29T15:51:05.102 INFO:teuthology.orchestra.run.smithi168.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-29T15:51:05.102 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_CODENAME=bionic 2020-11-29T15:51:05.102 INFO:teuthology.orchestra.run.smithi168.stdout:UBUNTU_CODENAME=bionic 2020-11-29T15:51:05.103 INFO:teuthology.lock.ops:Updating smithi168.front.sepia.ceph.com on lock server 2020-11-29T15:51:05.118 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: set os type and version on server 2020-11-29T15:51:05.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-29T15:51:05.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:51:05.193 INFO:teuthology.orchestra.run.smithi184:> true 2020-11-29T15:51:05.435 INFO:teuthology.orchestra.run.smithi184:> uname -m 2020-11-29T15:51:05.483 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2020-11-29T15:51:05.484 INFO:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2020-11-29T15:51:05.531 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu" 2020-11-29T15:51:05.531 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-29T15:51:05.532 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu 2020-11-29T15:51:05.532 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian 2020-11-29T15:51:05.532 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-29T15:51:05.532 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="18.04" 2020-11-29T15:51:05.533 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-29T15:51:05.533 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-29T15:51:05.533 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-29T15:51:05.533 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-29T15:51:05.533 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=bionic 2020-11-29T15:51:05.534 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=bionic 2020-11-29T15:51:05.534 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2020-11-29T15:51:05.548 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-29T15:51:05.579 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi168.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdrYZbb8MPSUJwpZ8Q7VVCJhkVlWIs7odbYMnfk3AU5J446G411sRshkndZuW7PE3fXPGJD8BInXKRBhBSy+nkRiv9n/om6poKcIG5Yv1SdpNdOtxTpOXzBjpOeYYLVHrUpq/cza4Nh+IBhyhavgDiU5L0pP215lQHswHSTZPAkTMzNuWQQ2QhjV+d9kzDSfggdzVyU9Xj4q3SE+MQU6Aptm3J/Qhrxj11C8hUB2LYJGFFqKw8lWW50ylVaCByHuuLLmQhHRhl5Ysgpe5Rnnpk7GF10RN3yvQxmeW6XEl1ZrKGdk232VZWx7BGHW3+VF8DiLW6t0umenxaKPyTTX8V smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDadENPl2I5HwceuElxWd2qNVQqbzVzle1/SaPOx61ShU45ELWegCqyHgUkO4iCRopi05bspLcoF0hyQ7DncvrWAcmlw1mimGxjM+6UgakidoKNBba9EZ5FTG2O9nEXoNFYIe5DXoA+/QcrShkDQnTKWuKsb41jXhS1UrPNCK+wEZ98w5ClbEL8YMMMFdH6LNGyizsXvmFWKK4VLPhWlqGVa2zi+P8P28XoUn3fI8OvQ8DTnlDhRZAbuduRfopcjIKERfn2SU2z+0dmj8bUoTL5vS809ClPLFR2xTprUkI/dmRJE4JGUb+E/ns79IIn+AG2FCys0qZ0YGaLrmTzPPXT 2020-11-29T15:51:05.580 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-29T15:51:05.628 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-29T15:51:05.638 INFO:teuthology.task.internal:Saving configuration 2020-11-29T15:51:05.653 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-29T15:51:05.719 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-29T15:51:05.736 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-29 15:45:24.950353', 'description': '/home/teuthworker/archive/mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi/5665427', 'locked_by': 'scheduled_mkogan@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAEuuvDD8jsNmNqIMS3PkTVV7+ixjE6qNVRYxDhJzMTZ', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi168.front.sepia.ceph.com'} 2020-11-29T15:51:05.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-29 15:45:24.950375', 'description': '/home/teuthworker/archive/mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi/5665427', 'locked_by': 'scheduled_mkogan@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBhRzJNZT1nfTdMIyBnUEXOqLof8im2qQzyLN8F8OiF8', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi184.front.sepia.ceph.com'} 2020-11-29T15:51:05.751 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-29T15:51:05.870 INFO:teuthology.task.internal:roles: ubuntu@smithi168.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-11-29T15:51:05.870 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-11-29T15:51:05.870 INFO:teuthology.run_tasks:Running task console_log... 2020-11-29T15:51:06.039 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3845ef2840>, signals=[15]) 2020-11-29T15:51:06.040 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-29T15:51:06.077 INFO:teuthology.task.internal:Opening connections... 2020-11-29T15:51:06.077 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi168.front.sepia.ceph.com 2020-11-29T15:51:06.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:51:06.153 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2020-11-29T15:51:06.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:51:06.231 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-29T15:51:06.238 INFO:teuthology.orchestra.run.smithi168:> uname -m 2020-11-29T15:51:06.310 INFO:teuthology.orchestra.run.smithi168.stdout:x86_64 2020-11-29T15:51:06.311 INFO:teuthology.orchestra.run.smithi168:> cat /etc/os-release 2020-11-29T15:51:06.359 INFO:teuthology.orchestra.run.smithi168.stdout:NAME="Ubuntu" 2020-11-29T15:51:06.359 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-29T15:51:06.360 INFO:teuthology.orchestra.run.smithi168.stdout:ID=ubuntu 2020-11-29T15:51:06.360 INFO:teuthology.orchestra.run.smithi168.stdout:ID_LIKE=debian 2020-11-29T15:51:06.360 INFO:teuthology.orchestra.run.smithi168.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-29T15:51:06.360 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_ID="18.04" 2020-11-29T15:51:06.361 INFO:teuthology.orchestra.run.smithi168.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-29T15:51:06.361 INFO:teuthology.orchestra.run.smithi168.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-29T15:51:06.361 INFO:teuthology.orchestra.run.smithi168.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-29T15:51:06.361 INFO:teuthology.orchestra.run.smithi168.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-29T15:51:06.362 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_CODENAME=bionic 2020-11-29T15:51:06.362 INFO:teuthology.orchestra.run.smithi168.stdout:UBUNTU_CODENAME=bionic 2020-11-29T15:51:06.362 INFO:teuthology.lock.ops:Updating smithi168.front.sepia.ceph.com on lock server 2020-11-29T15:51:06.378 INFO:teuthology.orchestra.run.smithi184:> uname -m 2020-11-29T15:51:06.426 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2020-11-29T15:51:06.427 INFO:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2020-11-29T15:51:06.475 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu" 2020-11-29T15:51:06.475 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-29T15:51:06.475 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu 2020-11-29T15:51:06.476 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian 2020-11-29T15:51:06.476 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-29T15:51:06.476 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="18.04" 2020-11-29T15:51:06.476 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-29T15:51:06.476 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-29T15:51:06.477 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-29T15:51:06.477 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-29T15:51:06.477 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=bionic 2020-11-29T15:51:06.477 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=bionic 2020-11-29T15:51:06.478 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2020-11-29T15:51:06.493 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-29T15:51:06.512 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-29T15:51:06.522 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-29T15:51:06.523 INFO:teuthology.orchestra.run.smithi168:> test '!' -e /home/ubuntu/cephtest 2020-11-29T15:51:06.525 INFO:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2020-11-29T15:51:06.531 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-29T15:51:06.553 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-29T15:51:06.553 INFO:teuthology.orchestra.run.smithi168:> test -z $(ls -A /var/lib/ceph) 2020-11-29T15:51:06.571 INFO:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2020-11-29T15:51:06.586 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-29T15:51:06.630 INFO:teuthology.run_tasks:Running task kernel... 2020-11-29T15:51:06.648 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-11-29T15:51:06.648 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-29T15:51:06.649 INFO:teuthology.orchestra.run.smithi168:> uname -r 2020-11-29T15:51:06.655 INFO:teuthology.orchestra.run.smithi168.stdout:4.15.0-124-generic 2020-11-29T15:51:06.655 INFO:teuthology.task.kernel:Running kernel on smithi168: 4.15.0-124-generic 2020-11-29T15:51:06.655 INFO:teuthology.orchestra.run.smithi168:> sudo apt-get clean 2020-11-29T15:51:06.773 INFO:teuthology.orchestra.run.smithi168:> sudo apt-get update 2020-11-29T15:51:07.058 INFO:teuthology.orchestra.run.smithi168.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-29T15:51:07.085 INFO:teuthology.orchestra.run.smithi168.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-29T15:51:07.140 INFO:teuthology.orchestra.run.smithi168.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-29T15:51:07.447 INFO:teuthology.orchestra.run.smithi168.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-29T15:51:07.790 INFO:teuthology.orchestra.run.smithi168.stdout:Fetched 252 kB in 1s (321 kB/s) 2020-11-29T15:51:10.271 INFO:teuthology.orchestra.run.smithi168.stdout:Reading package lists... 2020-11-29T15:51:10.319 INFO:teuthology.orchestra.run.smithi168:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-29T15:51:10.398 INFO:teuthology.orchestra.run.smithi168.stdout:Reading package lists... 2020-11-29T15:51:10.636 INFO:teuthology.orchestra.run.smithi168.stdout:Building dependency tree... 2020-11-29T15:51:10.637 INFO:teuthology.orchestra.run.smithi168.stdout:Reading state information... 2020-11-29T15:51:10.715 INFO:teuthology.orchestra.run.smithi168.stderr:E: Unable to locate package linux-image-current-generic 2020-11-29T15:51:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-29T15:51:10.718 INFO:teuthology.orchestra.run.smithi168:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-29T15:51:10.799 INFO:teuthology.orchestra.run.smithi168.stdout:Reading package lists... 2020-11-29T15:51:11.048 INFO:teuthology.orchestra.run.smithi168.stdout:Building dependency tree... 2020-11-29T15:51:11.049 INFO:teuthology.orchestra.run.smithi168.stdout:Reading state information... 2020-11-29T15:51:11.253 INFO:teuthology.orchestra.run.smithi168.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-29T15:51:11.253 INFO:teuthology.orchestra.run.smithi168.stdout:linux-image-generic set to manually installed. 2020-11-29T15:51:11.254 INFO:teuthology.orchestra.run.smithi168.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2020-11-29T15:51:11.255 INFO:teuthology.orchestra.run.smithi168:> dpkg -s linux-image-generic 2020-11-29T15:51:11.277 INFO:teuthology.orchestra.run.smithi168.stdout:Package: linux-image-generic 2020-11-29T15:51:11.277 INFO:teuthology.orchestra.run.smithi168.stdout:Status: install ok installed 2020-11-29T15:51:11.277 INFO:teuthology.orchestra.run.smithi168.stdout:Priority: optional 2020-11-29T15:51:11.278 INFO:teuthology.orchestra.run.smithi168.stdout:Section: kernel 2020-11-29T15:51:11.278 INFO:teuthology.orchestra.run.smithi168.stdout:Installed-Size: 16 2020-11-29T15:51:11.278 INFO:teuthology.orchestra.run.smithi168.stdout:Maintainer: Ubuntu Kernel Team 2020-11-29T15:51:11.278 INFO:teuthology.orchestra.run.smithi168.stdout:Architecture: amd64 2020-11-29T15:51:11.279 INFO:teuthology.orchestra.run.smithi168.stdout:Source: linux-meta 2020-11-29T15:51:11.279 INFO:teuthology.orchestra.run.smithi168.stdout:Version: 4.15.0.124.111 2020-11-29T15:51:11.279 INFO:teuthology.orchestra.run.smithi168.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-29T15:51:11.279 INFO:teuthology.orchestra.run.smithi168.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-29T15:51:11.280 INFO:teuthology.orchestra.run.smithi168.stdout:Recommends: thermald 2020-11-29T15:51:11.280 INFO:teuthology.orchestra.run.smithi168.stdout:Description: Generic Linux kernel image 2020-11-29T15:51:11.280 INFO:teuthology.orchestra.run.smithi168.stdout: This package will always depend on the latest generic kernel image 2020-11-29T15:51:11.280 INFO:teuthology.orchestra.run.smithi168.stdout: available. 2020-11-29T15:51:11.280 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-29T15:51:11.281 INFO:teuthology.orchestra.run.smithi184:> uname -r 2020-11-29T15:51:11.287 INFO:teuthology.orchestra.run.smithi184.stdout:4.15.0-124-generic 2020-11-29T15:51:11.287 INFO:teuthology.task.kernel:Running kernel on smithi184: 4.15.0-124-generic 2020-11-29T15:51:11.288 INFO:teuthology.orchestra.run.smithi184:> sudo apt-get clean 2020-11-29T15:51:11.494 INFO:teuthology.orchestra.run.smithi184:> sudo apt-get update 2020-11-29T15:51:11.637 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-29T15:51:11.754 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-29T15:51:11.833 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-29T15:51:12.123 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-29T15:51:12.483 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 252 kB in 1s (345 kB/s) 2020-11-29T15:51:14.856 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-11-29T15:51:14.904 INFO:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-29T15:51:14.981 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-11-29T15:51:15.238 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2020-11-29T15:51:15.238 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2020-11-29T15:51:15.322 INFO:teuthology.orchestra.run.smithi184.stderr:E: Unable to locate package linux-image-current-generic 2020-11-29T15:51:15.323 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-29T15:51:15.324 INFO:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-29T15:51:15.404 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-11-29T15:51:15.673 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2020-11-29T15:51:15.673 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2020-11-29T15:51:15.893 INFO:teuthology.orchestra.run.smithi184.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-29T15:51:15.893 INFO:teuthology.orchestra.run.smithi184.stdout:linux-image-generic set to manually installed. 2020-11-29T15:51:15.893 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2020-11-29T15:51:15.894 INFO:teuthology.orchestra.run.smithi184:> dpkg -s linux-image-generic 2020-11-29T15:51:15.916 INFO:teuthology.orchestra.run.smithi184.stdout:Package: linux-image-generic 2020-11-29T15:51:15.917 INFO:teuthology.orchestra.run.smithi184.stdout:Status: install ok installed 2020-11-29T15:51:15.917 INFO:teuthology.orchestra.run.smithi184.stdout:Priority: optional 2020-11-29T15:51:15.917 INFO:teuthology.orchestra.run.smithi184.stdout:Section: kernel 2020-11-29T15:51:15.917 INFO:teuthology.orchestra.run.smithi184.stdout:Installed-Size: 16 2020-11-29T15:51:15.918 INFO:teuthology.orchestra.run.smithi184.stdout:Maintainer: Ubuntu Kernel Team 2020-11-29T15:51:15.918 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: amd64 2020-11-29T15:51:15.918 INFO:teuthology.orchestra.run.smithi184.stdout:Source: linux-meta 2020-11-29T15:51:15.918 INFO:teuthology.orchestra.run.smithi184.stdout:Version: 4.15.0.124.111 2020-11-29T15:51:15.919 INFO:teuthology.orchestra.run.smithi184.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-29T15:51:15.919 INFO:teuthology.orchestra.run.smithi184.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-29T15:51:15.919 INFO:teuthology.orchestra.run.smithi184.stdout:Recommends: thermald 2020-11-29T15:51:15.919 INFO:teuthology.orchestra.run.smithi184.stdout:Description: Generic Linux kernel image 2020-11-29T15:51:15.919 INFO:teuthology.orchestra.run.smithi184.stdout: This package will always depend on the latest generic kernel image 2020-11-29T15:51:15.920 INFO:teuthology.orchestra.run.smithi184.stdout: available. 2020-11-29T15:51:15.920 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-29T15:51:15.920 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-11-29T15:51:15.921 INFO:teuthology.orchestra.run.smithi168:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-29T15:51:16.245 INFO:teuthology.orchestra.run.smithi168.stdout:ttyS1 2020-11-29T15:51:16.266 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-11-29T15:51:16.267 INFO:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-29T15:51:16.286 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2020-11-29T15:51:16.314 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-29T15:51:16.325 INFO:teuthology.task.internal:Creating test directory... 2020-11-29T15:51:16.325 INFO:teuthology.orchestra.run.smithi168:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-29T15:51:16.327 INFO:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-29T15:51:16.362 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-29T15:51:16.372 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-29T15:51:16.389 INFO:teuthology.task.internal:Creating archive directory... 2020-11-29T15:51:16.390 INFO:teuthology.orchestra.run.smithi168:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-29T15:51:16.392 INFO:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-29T15:51:16.418 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-29T15:51:16.428 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-29T15:51:16.429 INFO:teuthology.orchestra.run.smithi168:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-29T15:51:16.446 INFO:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-29T15:51:16.469 INFO:teuthology.orchestra.run.smithi168.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-29T15:51:16.477 INFO:teuthology.orchestra.run.smithi168.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-29T15:51:16.483 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-29T15:51:16.490 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-29T15:51:16.491 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-29T15:51:16.501 INFO:teuthology.task.internal:Configuring sudo... 2020-11-29T15:51:16.502 INFO:teuthology.orchestra.run.smithi168:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-29T15:51:16.522 INFO:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-29T15:51:16.547 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-29T15:51:16.558 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-29T15:51:16.559 INFO:teuthology.orchestra.run.smithi168:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-29T15:51:16.578 INFO:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-29T15:51:16.594 INFO:teuthology.orchestra.run.smithi168:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-29T15:51:16.630 INFO:teuthology.orchestra.run.smithi168:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-29T15:51:16.678 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:51:16.679 INFO:teuthology.orchestra.run.smithi168:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-29T15:51:16.724 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-29T15:51:16.742 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:51:16.742 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:51:16.743 INFO:teuthology.orchestra.run.smithi168.stderr:152 bytes copied, 3.2385e-05 s, 4.7 MB/s 2020-11-29T15:51:16.744 INFO:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-29T15:51:16.753 INFO:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-29T15:51:16.802 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:51:16.803 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-29T15:51:16.848 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-29T15:51:16.871 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:51:16.871 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:51:16.872 INFO:teuthology.orchestra.run.smithi184.stderr:152 bytes copied, 3.0047e-05 s, 5.1 MB/s 2020-11-29T15:51:16.873 INFO:teuthology.orchestra.run.smithi168:> sudo service rsyslog restart 2020-11-29T15:51:16.875 INFO:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2020-11-29T15:51:16.957 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-29T15:51:16.966 INFO:teuthology.task.internal:Starting timer... 2020-11-29T15:51:16.966 INFO:teuthology.run_tasks:Running task pcp... 2020-11-29T15:51:16.986 INFO:teuthology.run_tasks:Running task selinux... 2020-11-29T15:51:17.000 DEBUG:teuthology.task.selinux:Excluding smithi168: OS 'ubuntu' does not support SELinux 2020-11-29T15:51:17.000 DEBUG:teuthology.task.selinux:Excluding smithi184: OS 'ubuntu' does not support SELinux 2020-11-29T15:51:17.000 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-29T15:51:17.001 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-11-29T15:51:17.001 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-29T15:51:17.001 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-29T15:51:17.010 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-11-29T15:51:17.011 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-29T15:51:17.022 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-29T15:51:17.024 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi168.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-29T15:51:17.330 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-29T15:51:18.182 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-29T15:51:20.700 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-29T15:51:20.719 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:20 +0000 (0:00:00.068) 0:00:00.068 ******* 2020-11-29T15:51:20.732 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:20 +0000 (0:00:00.012) 0:00:00.081 ******* 2020-11-29T15:51:22.333 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi168.front.sepia.ceph.com] 2020-11-29T15:51:22.407 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:22 +0000 (0:00:01.675) 0:00:01.757 ******* 2020-11-29T15:51:22.458 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] 2020-11-29T15:51:22.533 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:22 +0000 (0:00:00.125) 0:00:01.882 ******* 2020-11-29T15:51:22.652 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-11-29T15:51:22.654 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-29T15:51:22.693 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:22 +0000 (0:00:00.160) 0:00:02.043 ******* 2020-11-29T15:51:22.725 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:22 +0000 (0:00:00.031) 0:00:02.074 ******* 2020-11-29T15:51:23.133 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-29T15:51:23.135 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-29T15:51:23.169 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:23 +0000 (0:00:00.443) 0:00:02.518 ******* 2020-11-29T15:51:23.174 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-29T15:51:23.210 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:23 +0000 (0:00:00.041) 0:00:02.560 ******* 2020-11-29T15:51:23.474 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-29T15:51:23.475 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.065356", "end": "2020-11-29 15:51:23.030544", "rc": 0, "start": "2020-11-29 15:51:22.965188", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-29T15:51:23.509 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:23 +0000 (0:00:00.298) 0:00:02.858 ******* 2020-11-29T15:51:23.514 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.076516", "end": "2020-11-29 15:51:23.233092", "rc": 0, "start": "2020-11-29 15:51:23.156576", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-29T15:51:23.551 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:23 +0000 (0:00:00.041) 0:00:02.900 ******* 2020-11-29T15:51:23.763 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-11-29T15:51:23.764 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-29T15:51:23.799 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:23 +0000 (0:00:00.247) 0:00:03.148 ******* 2020-11-29T15:51:23.830 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:23 +0000 (0:00:00.031) 0:00:03.180 ******* 2020-11-29T15:51:24.508 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-29T15:51:24.510 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1606665083.8430607-226525378525624/source", "state": "file", "uid": 0} changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1606665083.8126163-176650217017051/source", "state": "file", "uid": 0} 2020-11-29T15:51:24.549 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:24 +0000 (0:00:00.718) 0:00:03.898 ******* 2020-11-29T15:51:24.581 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:24 +0000 (0:00:00.031) 0:00:03.930 ******* 2020-11-29T15:51:24.834 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-29T15:51:24.836 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-29T15:51:24.837 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2020-11-29T15:51:24.839 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2020-11-29T15:51:24.841 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-29T15:51:24.971 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-29T15:51:24.978 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-29T15:51:25.125 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:25 +0000 (0:00:00.544) 0:00:04.474 ******* 2020-11-29T15:51:25.156 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:25 +0000 (0:00:00.031) 0:00:04.505 ******* 2020-11-29T15:51:25.860 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi168.front.sepia.ceph.com] 2020-11-29T15:51:25.908 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] 2020-11-29T15:51:25.952 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-29T15:51:25.954 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:25 +0000 (0:00:00.795) 0:00:05.301 ******* 2020-11-29T15:51:26.023 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.070) 0:00:05.372 ******* 2020-11-29T15:51:26.315 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-11-29T15:51:26.316 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.149658", "end": "2020-11-29 15:51:26.287230", "rc": 0, "start": "2020-11-29 15:51:26.137572", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-29T15:51:26.353 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.329) 0:00:05.702 ******* 2020-11-29T15:51:26.381 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.162916", "end": "2020-11-29 15:51:26.354546", "rc": 0, "start": "2020-11-29 15:51:26.191630", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-29T15:51:26.418 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.064) 0:00:05.767 ******* 2020-11-29T15:51:26.496 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-29T15:51:26.498 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009076", "end": "2020-11-29 15:51:26.244806", "failed_when_result": false, "rc": 0, "start": "2020-11-29 15:51:26.235730", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-29T15:51:26.534 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.116) 0:00:05.883 ******* 2020-11-29T15:51:26.552 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.007602", "end": "2020-11-29 15:51:26.107933", "failed_when_result": false, "rc": 0, "start": "2020-11-29 15:51:26.100331", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-29T15:51:26.594 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.060) 0:00:05.943 ******* 2020-11-29T15:51:26.603 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-29T15:51:26.640 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.046) 0:00:05.990 ******* 2020-11-29T15:51:26.647 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-29T15:51:26.682 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.041) 0:00:06.031 ******* 2020-11-29T15:51:26.687 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-29T15:51:26.689 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:26.724 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.041) 0:00:06.073 ******* 2020-11-29T15:51:26.730 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:26.765 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.041) 0:00:06.114 ******* 2020-11-29T15:51:26.772 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-11-29T15:51:26.773 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:26.807 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.041) 0:00:06.156 ******* 2020-11-29T15:51:26.813 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:26.848 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.041) 0:00:06.197 ******* 2020-11-29T15:51:26.853 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-29T15:51:26.855 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:26.889 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.040) 0:00:06.238 ******* 2020-11-29T15:51:26.895 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:26.934 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.045) 0:00:06.283 ******* 2020-11-29T15:51:26.940 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-29T15:51:26.942 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:26.982 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:26 +0000 (0:00:00.044) 0:00:06.328 ******* 2020-11-29T15:51:26.986 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.024 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:27 +0000 (0:00:00.044) 0:00:06.373 ******* 2020-11-29T15:51:27.030 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-29T15:51:27.032 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.065 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:27 +0000 (0:00:00.041) 0:00:06.414 ******* 2020-11-29T15:51:27.071 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.110 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:27 +0000 (0:00:00.044) 0:00:06.459 ******* 2020-11-29T15:51:27.126 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-29T15:51:27.128 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ 2020-11-29T15:51:27.130 INFO:teuthology.task.ansible.out:ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.140 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.142 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.151 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.153 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.162 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.164 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.174 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.176 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvS 2020-11-29T15:51:27.178 INFO:teuthology.task.ansible.out:KlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.187 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.189 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440 2020-11-29T15:51:27.190 INFO:teuthology.task.ansible.out:e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.199 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.201 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.202 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.202 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.203 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f 2020-11-29T15:51:27.205 INFO:teuthology.task.ansible.out:43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.215 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.217 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk728 2020-11-29T15:51:27.219 INFO:teuthology.task.ansible.out:1ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.228 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.229 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric 2020-11-29T15:51:27.231 INFO:teuthology.task.ansible.out:TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5S 2020-11-29T15:51:27.233 INFO:teuthology.task.ansible.out:Gs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.241 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.243 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f2 2020-11-29T15:51:27.245 INFO:teuthology.task.ansible.out:7af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.254 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.256 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedo 2020-11-29T15:51:27.257 INFO:teuthology.task.ansible.out:ragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.267 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.268 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.279 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.281 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.282 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icA 2020-11-29T15:51:27.283 INFO:teuthology.task.ansible.out:tDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.292 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.294 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.304 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.305 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.307 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.307 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"na 2020-11-29T15:51:27.309 INFO:teuthology.task.ansible.out:me": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.319 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.320 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.321 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.332 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.333 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.335 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.346 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.347 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.348 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.359 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.360 INFO:teuthology.task.ansible.out: 2020-11-29T15:51:27.362 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.363 INFO:teuthology.task.ansible.out: 2020-11-29T15:51:27.364 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.366 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.378 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.379 INFO:teuthology.task.ansible.out: skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.381 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.382 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.393 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.395 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.395 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.407 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.408 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.418 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.420 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.420 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a95 2020-11-29T15:51:27.422 INFO:teuthology.task.ansible.out:3dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.432 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.433 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.443 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.445 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.446 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.447 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.458 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.460 INFO:teuthology.task.ansible.out: skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.460 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.471 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.473 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.484 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.485 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.496 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.498 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.509 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.510 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.511 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.521 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.523 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.534 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.536 INFO:teuthology.task.ansible.out: skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.546 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.547 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.547 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.548 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.548 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.559 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.561 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.561 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.572 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.574 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.585 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.587 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.588 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rish 2020-11-29T15:51:27.590 INFO:teuthology.task.ansible.out:abh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.598 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.600 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.611 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.612 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.613 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.613 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": { 2020-11-29T15:51:27.615 INFO:teuthology.task.ansible.out:"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.624 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.625 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.636 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.638 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.638 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.649 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.651 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.661 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.664 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.676 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.678 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.678 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.680 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.680 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.681 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.692 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.694 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.704 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.705 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.706 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.716 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.718 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.729 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.730 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.731 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.731 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": 2020-11-29T15:51:27.733 INFO:teuthology.task.ansible.out: {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.742 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.744 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.754 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.756 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.756 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.767 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.769 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.779 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.780 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.780 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.782 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.792 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.794 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.795 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.805 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.808 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.818 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.820 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.820 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.831 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.832 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.850 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.851 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.853 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.855 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.899 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:27 +0000 (0:00:00.789) 0:00:07.248 ******* 2020-11-29T15:51:27.909 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.910 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.911 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.914 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.914 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.916 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.916 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.918 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.920 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:27.936 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-29T15:51:27.971 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:27 +0000 (0:00:00.071) 0:00:07.320 ******* 2020-11-29T15:51:28.004 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.033) 0:00:07.353 ******* 2020-11-29T15:51:28.010 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-29T15:51:28.011 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.050 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.046) 0:00:07.399 ******* 2020-11-29T15:51:28.057 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-29T15:51:28.093 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.043) 0:00:07.443 ******* 2020-11-29T15:51:28.099 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-29T15:51:28.101 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.136 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.042) 0:00:07.485 ******* 2020-11-29T15:51:28.142 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-29T15:51:28.144 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.185 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.049) 0:00:07.534 ******* 2020-11-29T15:51:28.191 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-29T15:51:28.193 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.228 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.042) 0:00:07.577 ******* 2020-11-29T15:51:28.235 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-29T15:51:28.237 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.270 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.042) 0:00:07.619 ******* 2020-11-29T15:51:28.276 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-29T15:51:28.278 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.314 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.043) 0:00:07.663 ******* 2020-11-29T15:51:28.320 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-29T15:51:28.322 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.356 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.042) 0:00:07.705 ******* 2020-11-29T15:51:28.361 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-11-29T15:51:28.363 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.397 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.041) 0:00:07.747 ******* 2020-11-29T15:51:28.403 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-29T15:51:28.405 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.441 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.043) 0:00:07.791 ******* 2020-11-29T15:51:28.447 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-29T15:51:28.449 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.449 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.451 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.461 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.463 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.498 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.056) 0:00:07.847 ******* 2020-11-29T15:51:28.504 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-29T15:51:28.506 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.506 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.508 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.520 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.531 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.533 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.544 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.546 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.556 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.558 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.568 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.570 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.581 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.583 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.594 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.596 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.606 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.608 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.618 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.620 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.621 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.622 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.623 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.633 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.635 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.646 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.647 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.658 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.659 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.660 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.661 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.705 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.207) 0:00:08.054 ******* 2020-11-29T15:51:28.713 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.714 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.715 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.716 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.716 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.717 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.717 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.718 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.729 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.740 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.742 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.751 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.753 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.762 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.764 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.773 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:28.813 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:28 +0000 (0:00:00.107) 0:00:08.162 ******* 2020-11-29T15:51:29.245 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-29T15:51:29.246 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-29T15:51:29.281 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.468) 0:00:08.630 ******* 2020-11-29T15:51:29.288 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-29T15:51:29.329 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.047) 0:00:08.678 ******* 2020-11-29T15:51:29.335 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-29T15:51:29.337 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => { "msg": "Host smithi168.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-29T15:51:29.376 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.047) 0:00:08.725 ******* 2020-11-29T15:51:29.382 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => { "msg": "Host smithi184.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-29T15:51:29.444 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.067) 0:00:08.793 ******* 2020-11-29T15:51:29.449 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-29T15:51:29.451 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-29T15:51:29.486 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.042) 0:00:08.835 ******* 2020-11-29T15:51:29.492 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-29T15:51:29.528 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.041) 0:00:08.877 ******* 2020-11-29T15:51:29.533 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-29T15:51:29.535 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.569 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.041) 0:00:08.919 ******* 2020-11-29T15:51:29.576 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.612 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.042) 0:00:08.961 ******* 2020-11-29T15:51:29.617 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-29T15:51:29.619 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.654 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.042) 0:00:09.003 ******* 2020-11-29T15:51:29.660 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.697 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.042) 0:00:09.046 ******* 2020-11-29T15:51:29.703 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-29T15:51:29.704 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.741 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.043) 0:00:09.090 ******* 2020-11-29T15:51:29.747 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.783 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.042) 0:00:09.132 ******* 2020-11-29T15:51:29.789 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-29T15:51:29.790 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.825 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.041) 0:00:09.174 ******* 2020-11-29T15:51:29.832 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.866 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.041) 0:00:09.215 ******* 2020-11-29T15:51:29.872 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-29T15:51:29.874 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.908 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.041) 0:00:09.257 ******* 2020-11-29T15:51:29.914 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.952 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.044) 0:00:09.301 ******* 2020-11-29T15:51:29.958 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-29T15:51:29.960 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:29.994 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:29 +0000 (0:00:00.041) 0:00:09.343 ******* 2020-11-29T15:51:30.001 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.035 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.041) 0:00:09.384 ******* 2020-11-29T15:51:30.041 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-29T15:51:30.042 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.079 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.043) 0:00:09.428 ******* 2020-11-29T15:51:30.085 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.121 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.042) 0:00:09.470 ******* 2020-11-29T15:51:30.127 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-29T15:51:30.129 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.162 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.041) 0:00:09.512 ******* 2020-11-29T15:51:30.168 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.204 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.041) 0:00:09.554 ******* 2020-11-29T15:51:30.209 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-29T15:51:30.211 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:30.247 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.042) 0:00:09.596 ******* 2020-11-29T15:51:30.253 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:30.254 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:30.288 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.041) 0:00:09.637 ******* 2020-11-29T15:51:30.295 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-29T15:51:30.297 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.330 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.042) 0:00:09.680 ******* 2020-11-29T15:51:30.337 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.373 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.042) 0:00:09.722 ******* 2020-11-29T15:51:30.380 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-29T15:51:30.382 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.422 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.049) 0:00:09.771 ******* 2020-11-29T15:51:30.430 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.467 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.044) 0:00:09.816 ******* 2020-11-29T15:51:30.473 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-29T15:51:30.474 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:30.509 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.041) 0:00:09.858 ******* 2020-11-29T15:51:30.515 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:30.559 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.050) 0:00:09.908 ******* 2020-11-29T15:51:30.567 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-29T15:51:30.569 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.624 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.064) 0:00:09.973 ******* 2020-11-29T15:51:30.631 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.665 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.041) 0:00:10.015 ******* 2020-11-29T15:51:30.671 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-29T15:51:30.673 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:30.708 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.042) 0:00:10.057 ******* 2020-11-29T15:51:30.714 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:30.749 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.041) 0:00:10.098 ******* 2020-11-29T15:51:30.753 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-29T15:51:30.755 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.790 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.041) 0:00:10.139 ******* 2020-11-29T15:51:30.797 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.833 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.042) 0:00:10.182 ******* 2020-11-29T15:51:30.839 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-29T15:51:30.841 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.875 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.042) 0:00:10.224 ******* 2020-11-29T15:51:30.882 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.918 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.043) 0:00:10.267 ******* 2020-11-29T15:51:30.926 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-29T15:51:30.927 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:30.962 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:30 +0000 (0:00:00.043) 0:00:10.311 ******* 2020-11-29T15:51:30.968 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.004 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.042) 0:00:10.353 ******* 2020-11-29T15:51:31.012 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-29T15:51:31.013 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.048 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.044) 0:00:10.397 ******* 2020-11-29T15:51:31.054 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.090 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.042) 0:00:10.439 ******* 2020-11-29T15:51:31.097 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-29T15:51:31.099 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:31.132 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.042) 0:00:10.482 ******* 2020-11-29T15:51:31.139 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:31.194 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.061) 0:00:10.543 ******* 2020-11-29T15:51:31.200 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-29T15:51:31.202 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.236 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.041) 0:00:10.585 ******* 2020-11-29T15:51:31.243 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.278 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.042) 0:00:10.627 ******* 2020-11-29T15:51:31.284 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-29T15:51:31.285 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.320 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.042) 0:00:10.669 ******* 2020-11-29T15:51:31.326 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.362 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.042) 0:00:10.712 ******* 2020-11-29T15:51:31.368 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-29T15:51:31.370 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.405 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.042) 0:00:10.754 ******* 2020-11-29T15:51:31.411 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.448 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.043) 0:00:10.797 ******* 2020-11-29T15:51:31.454 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-29T15:51:31.455 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.490 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.041) 0:00:10.839 ******* 2020-11-29T15:51:31.496 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.531 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.041) 0:00:10.880 ******* 2020-11-29T15:51:31.538 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-29T15:51:31.539 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.573 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.041) 0:00:10.922 ******* 2020-11-29T15:51:31.579 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.618 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.045) 0:00:10.968 ******* 2020-11-29T15:51:31.625 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-29T15:51:31.626 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.676 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.057) 0:00:11.025 ******* 2020-11-29T15:51:31.683 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.721 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.045) 0:00:11.071 ******* 2020-11-29T15:51:31.727 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-29T15:51:31.729 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:31.767 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.046) 0:00:11.117 ******* 2020-11-29T15:51:31.775 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:31.823 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.055) 0:00:11.172 ******* 2020-11-29T15:51:31.830 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-29T15:51:31.831 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.887 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.064) 0:00:11.236 ******* 2020-11-29T15:51:31.897 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.933 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.045) 0:00:11.282 ******* 2020-11-29T15:51:31.939 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-29T15:51:31.940 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:31.979 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:31 +0000 (0:00:00.045) 0:00:11.328 ******* 2020-11-29T15:51:31.985 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.025 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.046) 0:00:11.374 ******* 2020-11-29T15:51:32.031 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-29T15:51:32.033 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.070 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.044) 0:00:11.419 ******* 2020-11-29T15:51:32.076 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.118 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.048) 0:00:11.467 ******* 2020-11-29T15:51:32.124 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.126 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.189 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.071) 0:00:11.538 ******* 2020-11-29T15:51:32.206 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.462 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.080) 0:00:11.619 ******* TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi168.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} Sunday 29 November 2020 15:51:32 +0000 (0:00:00.045) 0:00:11.665 ******* skipping: [smithi184.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} Sunday 29 November 2020 15:51:32 +0000 (0:00:00.043) 0:00:11.708 ******* TASK [common : Clean yum cache] ************************************************ skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Sunday 29 November 2020 15:51:32 +0000 (0:00:00.044) 0:00:11.752 ******* skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Sunday 29 November 2020 15:51:32 +0000 (0:00:00.043) 0:00:11.796 ******* 2020-11-29T15:51:32.598 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-29T15:51:32.600 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.015540", "end": "2020-11-29 15:51:32.163921", "rc": 0, "start": "2020-11-29 15:51:32.148381", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-29T15:51:32.636 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.189) 0:00:11.985 ******* 2020-11-29T15:51:32.642 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.083483", "end": "2020-11-29 15:51:32.376370", "rc": 0, "start": "2020-11-29 15:51:32.292887", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-29T15:51:32.686 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.050) 0:00:12.035 ******* 2020-11-29T15:51:32.692 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-29T15:51:32.693 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.728 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.042) 0:00:12.077 ******* 2020-11-29T15:51:32.735 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.784 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.056) 0:00:12.133 ******* 2020-11-29T15:51:32.793 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-29T15:51:32.795 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.831 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.046) 0:00:12.180 ******* 2020-11-29T15:51:32.837 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.874 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.042) 0:00:12.223 ******* 2020-11-29T15:51:32.880 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-29T15:51:32.881 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.916 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.042) 0:00:12.265 ******* 2020-11-29T15:51:32.922 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:32.957 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.041) 0:00:12.306 ******* 2020-11-29T15:51:32.963 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-29T15:51:32.964 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.001 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:32 +0000 (0:00:00.043) 0:00:12.350 ******* 2020-11-29T15:51:33.009 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.050 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.048) 0:00:12.399 ******* 2020-11-29T15:51:33.057 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-29T15:51:33.058 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.095 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.045) 0:00:12.445 ******* 2020-11-29T15:51:33.101 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.140 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.044) 0:00:12.489 ******* 2020-11-29T15:51:33.146 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-29T15:51:33.147 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.184 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.044) 0:00:12.533 ******* 2020-11-29T15:51:33.190 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.227 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.042) 0:00:12.576 ******* 2020-11-29T15:51:33.233 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-29T15:51:33.234 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.268 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.041) 0:00:12.618 ******* 2020-11-29T15:51:33.274 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.311 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.042) 0:00:12.661 ******* 2020-11-29T15:51:33.749 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-29T15:51:33.750 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.303184", "end": "2020-11-29 15:51:33.500393", "rc": 0, "start": "2020-11-29 15:51:33.197209", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:51:33.785 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.473) 0:00:13.134 ******* 2020-11-29T15:51:33.811 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-29T15:51:33.813 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:33.848 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.062) 0:00:13.197 ******* 2020-11-29T15:51:33.938 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.543747", "end": "2020-11-29 15:51:33.500361", "rc": 0, "start": "2020-11-29 15:51:32.956614", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:51:33.976 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:33 +0000 (0:00:00.128) 0:00:13.325 ******* 2020-11-29T15:51:34.003 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-29T15:51:34.005 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.026416", "end": "2020-11-29 15:51:33.756070", "rc": 0, "start": "2020-11-29 15:51:33.729654", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:34.041 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:34 +0000 (0:00:00.064) 0:00:13.390 ******* 2020-11-29T15:51:34.077 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:34 +0000 (0:00:00.036) 0:00:13.426 ******* 2020-11-29T15:51:34.233 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-29T15:51:34.235 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023829", "end": "2020-11-29 15:51:33.795879", "rc": 0, "start": "2020-11-29 15:51:33.772050", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:51:34.269 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:34 +0000 (0:00:00.191) 0:00:13.618 ******* 2020-11-29T15:51:37.197 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-29T15:51:37.199 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1606665067, "cache_updated": false, "changed": false} ok: [smithi184.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:51:37.233 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:37 +0000 (0:00:02.964) 0:00:16.583 ******* 2020-11-29T15:51:37.270 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:37 +0000 (0:00:00.036) 0:00:16.619 ******* 2020-11-29T15:51:38.115 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-29T15:51:38.116 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:51:38.158 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:38 +0000 (0:00:00.888) 0:00:17.507 ******* 2020-11-29T15:51:38.165 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:51:38.202 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:38 +0000 (0:00:00.042) 0:00:17.550 ******* 2020-11-29T15:51:38.206 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-29T15:51:38.208 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:38.242 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:38 +0000 (0:00:00.040) 0:00:17.591 ******* 2020-11-29T15:51:38.248 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:38.285 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:38 +0000 (0:00:00.043) 0:00:17.635 ******* 2020-11-29T15:51:38.521 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-29T15:51:38.523 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-29T15:51:38.560 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:38 +0000 (0:00:00.274) 0:00:17.909 ******* 2020-11-29T15:51:38.847 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-29T15:51:38.883 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:38 +0000 (0:00:00.323) 0:00:18.232 ******* 2020-11-29T15:51:39.376 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-29T15:51:39.378 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:51:39.413 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:39 +0000 (0:00:00.530) 0:00:18.762 ******* 2020-11-29T15:51:39.713 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:51:39.752 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:39 +0000 (0:00:00.339) 0:00:19.102 ******* 2020-11-29T15:51:39.760 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-29T15:51:39.762 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-29T15:51:40.003 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-29T15:51:40.067 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-29T15:51:40.254 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-29T15:51:40.293 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:40 +0000 (0:00:00.540) 0:00:19.643 ******* 2020-11-29T15:51:40.351 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-29T15:51:40.415 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-29T15:51:40.416 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-29T15:51:40.450 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:40 +0000 (0:00:00.156) 0:00:19.799 ******* 2020-11-29T15:51:40.604 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-29T15:51:40.606 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-29T15:51:40.642 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:40 +0000 (0:00:00.191) 0:00:19.991 ******* 2020-11-29T15:51:40.741 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-29T15:51:40.743 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2020-11-29T15:51:40.764 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-29T15:51:40.766 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-29T15:51:40.807 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:40 +0000 (0:00:00.164) 0:00:20.156 ******* 2020-11-29T15:51:41.001 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-29T15:51:41.003 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-29T15:51:41.092 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2020-11-29T15:51:41.259 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-29T15:51:41.298 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.490) 0:00:20.647 ******* 2020-11-29T15:51:41.336 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-29T15:51:41.338 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi184.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-29T15:51:41.381 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.082) 0:00:20.729 ******* 2020-11-29T15:51:41.407 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-29T15:51:41.409 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.445 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.064) 0:00:20.794 ******* 2020-11-29T15:51:41.472 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-29T15:51:41.474 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:41.513 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.067) 0:00:20.862 ******* 2020-11-29T15:51:41.550 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-29T15:51:41.552 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.561 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.572 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.574 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.610 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.097) 0:00:20.959 ******* 2020-11-29T15:51:41.616 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-29T15:51:41.618 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-29T15:51:41.661 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.050) 0:00:21.010 ******* 2020-11-29T15:51:41.667 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-29T15:51:41.668 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.708 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.046) 0:00:21.057 ******* 2020-11-29T15:51:41.714 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-29T15:51:41.716 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-29T15:51:41.752 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.044) 0:00:21.101 ******* 2020-11-29T15:51:41.789 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-29T15:51:41.791 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.831 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.079) 0:00:21.180 ******* 2020-11-29T15:51:41.858 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-29T15:51:41.859 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:41.894 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.062) 0:00:21.243 ******* 2020-11-29T15:51:41.933 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-29T15:51:41.935 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.944 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.955 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.957 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:41.997 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:41 +0000 (0:00:00.102) 0:00:21.346 ******* 2020-11-29T15:51:42.023 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-29T15:51:42.024 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:42.060 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:42 +0000 (0:00:00.063) 0:00:21.410 ******* 2020-11-29T15:51:42.605 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-29T15:51:42.607 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:51:42.642 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:42 +0000 (0:00:00.581) 0:00:21.991 ******* 2020-11-29T15:51:42.953 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-29T15:51:42.955 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-29T15:51:42.993 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:42 +0000 (0:00:00.350) 0:00:22.342 ******* 2020-11-29T15:51:43.001 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-29T15:51:43.002 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:51:43.040 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:43 +0000 (0:00:00.047) 0:00:22.390 ******* 2020-11-29T15:51:43.279 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-29T15:51:43.281 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-29T15:51:43.317 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:43 +0000 (0:00:00.276) 0:00:22.666 ******* 2020-11-29T15:51:43.333 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-29T15:51:43.335 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-11-29T15:51:43.371 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:43 +0000 (0:00:00.053) 0:00:22.720 ******* 2020-11-29T15:51:43.460 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-29T15:51:43.462 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-29T15:51:43.565 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-29T15:51:43.602 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:43 +0000 (0:00:00.231) 0:00:22.952 ******* 2020-11-29T15:51:43.629 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-29T15:51:43.631 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:43.668 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:43 +0000 (0:00:00.065) 0:00:23.017 ******* 2020-11-29T15:51:43.675 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-29T15:51:43.677 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-29T15:51:43.717 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:43 +0000 (0:00:00.048) 0:00:23.066 ******* 2020-11-29T15:51:43.723 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-29T15:51:43.725 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:43.760 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:43 +0000 (0:00:00.043) 0:00:23.109 ******* 2020-11-29T15:51:43.788 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-29T15:51:43.789 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:43.824 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:43 +0000 (0:00:00.064) 0:00:23.173 ******* 2020-11-29T15:51:43.851 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-29T15:51:43.853 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-29T15:51:43.958 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-29T15:51:44.006 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.182) 0:00:23.355 ******* 2020-11-29T15:51:44.036 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:44.074 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.068) 0:00:23.423 ******* 2020-11-29T15:51:44.104 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-29T15:51:44.105 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:44.152 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.077) 0:00:23.501 ******* 2020-11-29T15:51:44.158 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-29T15:51:44.160 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-29T15:51:44.196 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.044) 0:00:23.546 ******* 2020-11-29T15:51:44.203 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-29T15:51:44.205 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:44.240 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.043) 0:00:23.589 ******* 2020-11-29T15:51:44.552 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-29T15:51:44.554 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-29T15:51:44.595 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.354) 0:00:23.944 ******* 2020-11-29T15:51:44.809 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-29T15:51:44.810 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2020-11-29 15:50:31 UTC", "ActiveEnterTimestampMonotonic": "19258686", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target systemd-journald.socket -.mount var-run.mount local-fs.target sysinit.target network.target systemd-tmpfiles-setup.service time-sync.target system.slice basic.target nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2020-11-29 15:50:31 UTC", "AssertTimestampMonotonic": "19257598", "Before": "getty@tty1.service shutdown.target xdm.service plymouth-quit.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2020-11-29 15:50:31 UTC", "ConditionTimestampMonotonic": "19257598", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-29T15:51:44.812 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1343", "ExecMainStartTimestamp": "Sun 2020-11-29 15:50:31 UTC", "ExecMainStartTimestampMonotonic": "19258644", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2020-11-29 15:50:31 UTC", "InactiveExitTimestampMonotonic": "19258686", "InvocationID": "a354e7e95c934495ad8f2d53a4e56e7b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-11-29T15:51:44.813 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1343", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-11-29T15:51:44.815 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2020-11-29 15:50:31 UTC", "StateChangeTimestampMonotonic": "19258686", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2020-11-29 15:50:31 UTC", "WatchdogTimestampMonotonic": "19258685", "WatchdogUSec": "0"}} ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2020-11-29 15:49:58 UTC", "ActiveEnterTimestampMonotonic": "19277722", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service systemd-journald.socket nss-lookup.target basic.target network.target system.slice local-fs.target remote-fs.target var-run.mount -.mount s 2020-11-29T15:51:44.816 INFO:teuthology.task.ansible.out:ysinit.target time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2020-11-29 15:49:58 UTC", "AssertTimestampMonotonic": "19276972", "Before": "multi-user.target xdm.service plymouth-quit.service getty@tty1.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2020-11-29 15:49:58 UTC", "ConditionTimestampMonotonic": "19276972", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1500", "ExecMainStartTimestamp": "Sun 2020-11-29 15:49:58 UTC", "ExecMainStartTimestampMonotonic": 2020-11-29T15:51:44.818 INFO:teuthology.task.ansible.out:"19277683", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2020-11-29 15:49:58 UTC", "InactiveExitTimestampMonotonic": "19277722", "InvocationID": "ca1156d3c25a4ab4849158b241e2f026", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "102 2020-11-29T15:51:44.820 INFO:teuthology.task.ansible.out:4", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1500", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "Star 2020-11-29T15:51:44.821 INFO:teuthology.task.ansible.out:tLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2020-11-29 15:49:58 UTC", "StateChangeTimestampMonotonic": "19277722", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2020-11-29 15:49:58 UTC", "WatchdogTimestampMonotonic": "19277721", "WatchdogUSec": "0"}} 2020-11-29T15:51:44.849 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.254) 0:00:24.198 ******* 2020-11-29T15:51:44.881 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.032) 0:00:24.231 ******* 2020-11-29T15:51:44.887 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-29T15:51:44.889 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:44.924 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.042) 0:00:24.273 ******* 2020-11-29T15:51:44.936 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:44.970 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:44 +0000 (0:00:00.045) 0:00:24.319 ******* 2020-11-29T15:51:44.975 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-29T15:51:44.977 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.012 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.042) 0:00:24.361 ******* 2020-11-29T15:51:45.018 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.054 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.041) 0:00:24.403 ******* 2020-11-29T15:51:45.059 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-29T15:51:45.060 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.095 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.040) 0:00:24.444 ******* 2020-11-29T15:51:45.100 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.138 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.042) 0:00:24.487 ******* 2020-11-29T15:51:45.143 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-29T15:51:45.145 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.179 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.041) 0:00:24.528 ******* 2020-11-29T15:51:45.185 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.220 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.041) 0:00:24.570 ******* 2020-11-29T15:51:45.227 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-29T15:51:45.228 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.262 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.041) 0:00:24.611 ******* 2020-11-29T15:51:45.268 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.305 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.043) 0:00:24.654 ******* 2020-11-29T15:51:45.311 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-29T15:51:45.313 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.348 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.043) 0:00:24.698 ******* 2020-11-29T15:51:45.355 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.391 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.042) 0:00:24.740 ******* 2020-11-29T15:51:45.397 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-29T15:51:45.398 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.433 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.041) 0:00:24.782 ******* 2020-11-29T15:51:45.439 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.474 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.041) 0:00:24.824 ******* 2020-11-29T15:51:45.480 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-29T15:51:45.482 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.518 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.043) 0:00:24.867 ******* 2020-11-29T15:51:45.530 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.564 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.046) 0:00:24.914 ******* 2020-11-29T15:51:45.570 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-29T15:51:45.572 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.607 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.042) 0:00:24.956 ******* 2020-11-29T15:51:45.613 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.649 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.041) 0:00:24.998 ******* 2020-11-29T15:51:45.654 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-29T15:51:45.656 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.691 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.041) 0:00:25.040 ******* 2020-11-29T15:51:45.697 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:45.734 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.042) 0:00:25.083 ******* 2020-11-29T15:51:45.874 INFO:teuthology.task.ansible.out: TASK [common : Check for podman] *********************************************** 2020-11-29T15:51:45.876 INFO:teuthology.task.ansible.out:fatal: [smithi184.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": "podman --version", "msg": "[Errno 2] No such file or directory", "rc": 2} ...ignoring 2020-11-29T15:51:45.911 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.177) 0:00:25.260 ******* 2020-11-29T15:51:45.937 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-29T15:51:45.939 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Check for podman] *********************************************** fatal: [smithi168.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": "podman --version", "msg": "[Errno 2] No such file or directory", "rc": 2} ...ignoring 2020-11-29T15:51:45.974 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:45 +0000 (0:00:00.062) 0:00:25.322 ******* 2020-11-29T15:51:46.005 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:46 +0000 (0:00:00.032) 0:00:25.354 ******* 2020-11-29T15:51:46.010 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-29T15:51:46.012 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:46.046 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:46 +0000 (0:00:00.040) 0:00:25.395 ******* 2020-11-29T15:51:50.485 INFO:teuthology.task.ansible.out: TASK [common : Check for docker] *********************************************** changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["docker", "--version"], "delta": "0:00:04.346550", "end": "2020-11-29 15:51:50.054858", "rc": 0, "start": "2020-11-29 15:51:45.708308", "stderr": "", "stderr_lines": [], "stdout": "Docker version 19.03.6, build 369ce74a3c", "stdout_lines": ["Docker version 19.03.6, build 369ce74a3c"]} 2020-11-29T15:51:50.521 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:50 +0000 (0:00:04.474) 0:00:29.870 ******* 2020-11-29T15:51:50.557 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-29T15:51:50.559 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000"}, "changed": false} 2020-11-29T15:51:50.600 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:50 +0000 (0:00:00.077) 0:00:29.947 ******* 2020-11-29T15:51:50.604 INFO:teuthology.task.ansible.out: TASK [common : Check for docker] *********************************************** 2020-11-29T15:51:50.606 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["docker", "--version"], "delta": "0:00:04.416124", "end": "2020-11-29 15:51:50.353385", "rc": 0, "start": "2020-11-29 15:51:45.937261", "stderr": "", "stderr_lines": [], "stdout": "Docker version 19.03.6, build 369ce74a3c", "stdout_lines": ["Docker version 19.03.6, build 369ce74a3c"]} 2020-11-29T15:51:50.660 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:50 +0000 (0:00:00.061) 0:00:30.009 ******* 2020-11-29T15:51:50.666 INFO:teuthology.task.ansible.out: TASK [common : Include encrypted variables] ************************************ 2020-11-29T15:51:50.667 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:50.707 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:50 +0000 (0:00:00.046) 0:00:30.056 ******* 2020-11-29T15:51:50.713 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-29T15:51:50.715 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000"}, "changed": false} 2020-11-29T15:51:50.756 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:50 +0000 (0:00:00.049) 0:00:30.105 ******* 2020-11-29T15:51:50.805 INFO:teuthology.task.ansible.out: TASK [common : Include encrypted variables] ************************************ 2020-11-29T15:51:50.807 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi168.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-29T15:51:50.842 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:50 +0000 (0:00:00.085) 0:00:30.191 ******* 2020-11-29T15:51:50.860 INFO:teuthology.task.ansible.out: TASK [common : Create /etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000] *** 2020-11-29T15:51:50.862 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2020-11-29T15:51:50.902 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:50 +0000 (0:00:00.059) 0:00:30.251 ******* 2020-11-29T15:51:50.983 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2020-11-29T15:51:51.019 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.117) 0:00:30.369 ******* 2020-11-29T15:51:51.225 INFO:teuthology.task.ansible.out: TASK [common : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2020-11-29T15:51:51.227 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1606665110.9321487-27635802143127/source", "state": "file", "uid": 0} 2020-11-29T15:51:51.266 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.246) 0:00:30.615 ******* 2020-11-29T15:51:51.293 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-29T15:51:51.295 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-29T15:51:51.331 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.065) 0:00:30.680 ******* 2020-11-29T15:51:51.349 INFO:teuthology.task.ansible.out: TASK [common : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2020-11-29T15:51:51.351 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1606665111.0466263-205015276397409/source", "state": "file", "uid": 0} 2020-11-29T15:51:51.394 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.063) 0:00:30.744 ******* 2020-11-29T15:51:51.403 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-29T15:51:51.403 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-29T15:51:51.446 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.051) 0:00:30.795 ******* 2020-11-29T15:51:51.452 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-29T15:51:51.454 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-29T15:51:51.489 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.043) 0:00:30.838 ******* 2020-11-29T15:51:51.495 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-29T15:51:51.497 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-29T15:51:51.534 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.044) 0:00:30.883 ******* 2020-11-29T15:51:51.540 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-29T15:51:51.542 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-29T15:51:51.583 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.048) 0:00:30.932 ******* 2020-11-29T15:51:51.589 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-29T15:51:51.591 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-29T15:51:51.626 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.043) 0:00:30.975 ******* 2020-11-29T15:51:51.752 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-29T15:51:51.754 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-29T15:51:51.803 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.176) 0:00:31.152 ******* 2020-11-29T15:51:51.840 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-29T15:51:51.842 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-29T15:51:51.879 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.076) 0:00:31.228 ******* 2020-11-29T15:51:51.885 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-29T15:51:51.887 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-29T15:51:51.925 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:51 +0000 (0:00:00.045) 0:00:31.274 ******* 2020-11-29T15:51:52.004 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-29T15:51:52.044 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.119) 0:00:31.393 ******* 2020-11-29T15:51:52.061 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-29T15:51:52.063 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "name": "ubuntu", "state": "present", "system": false} 2020-11-29T15:51:52.100 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.055) 0:00:31.449 ******* 2020-11-29T15:51:52.126 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-29T15:51:52.128 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:52.164 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.063) 0:00:31.513 ******* 2020-11-29T15:51:52.181 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-29T15:51:52.183 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "name": "ubuntu", "state": "present", "system": false} 2020-11-29T15:51:52.222 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.058) 0:00:31.571 ******* 2020-11-29T15:51:52.231 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-29T15:51:52.233 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:52.278 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.056) 0:00:31.628 ******* 2020-11-29T15:51:52.287 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-29T15:51:52.289 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:52.326 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.047) 0:00:31.676 ******* 2020-11-29T15:51:52.354 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-29T15:51:52.356 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:52.388 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.062) 0:00:31.738 ******* 2020-11-29T15:51:52.428 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-29T15:51:52.430 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-29T15:51:52.467 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.078) 0:00:31.816 ******* 2020-11-29T15:51:52.547 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-29T15:51:52.588 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.121) 0:00:31.937 ******* 2020-11-29T15:51:52.879 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-29T15:51:52.881 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-29T15:51:52.927 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.339) 0:00:32.277 ******* 2020-11-29T15:51:52.964 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-29T15:51:52.966 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi184"}, "changed": false} 2020-11-29T15:51:53.000 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:52 +0000 (0:00:00.072) 0:00:32.349 ******* 2020-11-29T15:51:53.006 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-29T15:51:53.008 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-29T15:51:53.043 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:53 +0000 (0:00:00.043) 0:00:32.392 ******* 2020-11-29T15:51:53.070 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-29T15:51:53.072 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi168"}, "changed": false} 2020-11-29T15:51:53.106 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:53 +0000 (0:00:00.062) 0:00:32.455 ******* 2020-11-29T15:51:53.834 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-11-29T15:51:53.836 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi168", "ansible_hostname": "smithi168", "ansible_nodename": "smithi168"}, "changed": false, "name": "smithi168"} 2020-11-29T15:51:53.870 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:53 +0000 (0:00:00.763) 0:00:33.219 ******* 2020-11-29T15:51:53.918 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi184", "ansible_hostname": "smithi184", "ansible_nodename": "smithi184"}, "changed": false, "name": "smithi184"} 2020-11-29T15:51:53.961 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:53 +0000 (0:00:00.091) 0:00:33.310 ******* 2020-11-29T15:51:54.174 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-11-29T15:51:54.176 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-29T15:51:54.212 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:54 +0000 (0:00:00.250) 0:00:33.561 ******* 2020-11-29T15:51:54.260 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-29T15:51:54.310 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:54 +0000 (0:00:00.097) 0:00:33.659 ******* 2020-11-29T15:51:54.519 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-29T15:51:54.557 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:54 +0000 (0:00:00.247) 0:00:33.906 ******* 2020-11-29T15:51:54.617 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-29T15:51:54.656 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:54 +0000 (0:00:00.098) 0:00:34.005 ******* 2020-11-29T15:51:55.082 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-29T15:51:55.084 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-29T15:51:55.118 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.462) 0:00:34.467 ******* 2020-11-29T15:51:55.144 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-29T15:51:55.146 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.182 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.063) 0:00:34.531 ******* 2020-11-29T15:51:55.189 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-29T15:51:55.191 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-29T15:51:55.234 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.052) 0:00:34.583 ******* 2020-11-29T15:51:55.243 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-29T15:51:55.245 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.281 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.046) 0:00:34.630 ******* 2020-11-29T15:51:55.286 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-29T15:51:55.288 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.323 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.041) 0:00:34.672 ******* 2020-11-29T15:51:55.328 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-29T15:51:55.330 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.365 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.041) 0:00:34.714 ******* 2020-11-29T15:51:55.371 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-29T15:51:55.373 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.407 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.042) 0:00:34.757 ******* 2020-11-29T15:51:55.413 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-29T15:51:55.415 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.453 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.045) 0:00:34.802 ******* 2020-11-29T15:51:55.459 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-29T15:51:55.461 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.498 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.044) 0:00:34.847 ******* 2020-11-29T15:51:55.503 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-29T15:51:55.505 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.542 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.044) 0:00:34.891 ******* 2020-11-29T15:51:55.548 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-29T15:51:55.550 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.591 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.049) 0:00:34.940 ******* 2020-11-29T15:51:55.598 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-29T15:51:55.599 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.641 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.049) 0:00:34.990 ******* 2020-11-29T15:51:55.648 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-29T15:51:55.650 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.687 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.045) 0:00:35.036 ******* 2020-11-29T15:51:55.692 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-29T15:51:55.694 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.730 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.042) 0:00:35.079 ******* 2020-11-29T15:51:55.736 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-29T15:51:55.738 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.773 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.043) 0:00:35.122 ******* 2020-11-29T15:51:55.778 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-11-29T15:51:55.780 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.822 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.049) 0:00:35.172 ******* 2020-11-29T15:51:55.829 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-29T15:51:55.831 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.865 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.042) 0:00:35.214 ******* 2020-11-29T15:51:55.871 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-29T15:51:55.907 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.042) 0:00:35.256 ******* 2020-11-29T15:51:55.913 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-11-29T15:51:55.914 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:55.949 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.041) 0:00:35.298 ******* 2020-11-29T15:51:55.955 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-29T15:51:55.990 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:55 +0000 (0:00:00.041) 0:00:35.339 ******* 2020-11-29T15:51:55.996 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-29T15:51:56.032 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.041) 0:00:35.381 ******* 2020-11-29T15:51:56.038 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-29T15:51:56.074 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.042) 0:00:35.423 ******* 2020-11-29T15:51:56.080 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-29T15:51:56.118 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.043) 0:00:35.467 ******* 2020-11-29T15:51:56.124 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-11-29T15:51:56.126 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.162 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.044) 0:00:35.511 ******* 2020-11-29T15:51:56.168 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-29T15:51:56.205 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.043) 0:00:35.555 ******* 2020-11-29T15:51:56.210 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-29T15:51:56.212 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.246 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.040) 0:00:35.595 ******* 2020-11-29T15:51:56.253 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.288 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.041) 0:00:35.637 ******* 2020-11-29T15:51:56.293 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-29T15:51:56.295 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.331 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.043) 0:00:35.680 ******* 2020-11-29T15:51:56.337 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-11-29T15:51:56.339 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.374 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.043) 0:00:35.723 ******* 2020-11-29T15:51:56.379 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:51:56.381 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.430 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.056) 0:00:35.779 ******* 2020-11-29T15:51:56.436 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-29T15:51:56.439 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.476 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.045) 0:00:35.825 ******* 2020-11-29T15:51:56.483 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-29T15:51:56.484 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.521 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.045) 0:00:35.870 ******* 2020-11-29T15:51:56.529 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:51:56.530 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.563 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.042) 0:00:35.913 ******* 2020-11-29T15:51:56.569 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-29T15:51:56.571 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.606 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.042) 0:00:35.955 ******* 2020-11-29T15:51:56.612 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-29T15:51:56.613 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.648 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.042) 0:00:35.998 ******* 2020-11-29T15:51:56.654 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-29T15:51:56.656 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.690 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.041) 0:00:36.039 ******* 2020-11-29T15:51:56.695 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-29T15:51:56.697 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.731 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.041) 0:00:36.080 ******* 2020-11-29T15:51:56.736 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-29T15:51:56.738 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.776 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.044) 0:00:36.125 ******* 2020-11-29T15:51:56.782 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-29T15:51:56.784 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.816 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.040) 0:00:36.165 ******* 2020-11-29T15:51:56.823 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-29T15:51:56.824 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.856 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.039) 0:00:36.205 ******* 2020-11-29T15:51:56.862 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-29T15:51:56.863 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.896 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.039) 0:00:36.245 ******* 2020-11-29T15:51:56.901 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-29T15:51:56.903 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.936 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.039) 0:00:36.285 ******* 2020-11-29T15:51:56.941 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-29T15:51:56.943 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:56.975 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:56 +0000 (0:00:00.039) 0:00:36.324 ******* 2020-11-29T15:51:56.981 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-29T15:51:56.982 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.016 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:36.365 ******* 2020-11-29T15:51:57.022 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-29T15:51:57.024 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.059 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.042) 0:00:36.408 ******* 2020-11-29T15:51:57.064 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-29T15:51:57.066 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.100 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:36.449 ******* 2020-11-29T15:51:57.107 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-29T15:51:57.109 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.142 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:36.491 ******* 2020-11-29T15:51:57.147 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-29T15:51:57.149 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.183 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:36.532 ******* 2020-11-29T15:51:57.189 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-29T15:51:57.190 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.225 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.042) 0:00:36.574 ******* 2020-11-29T15:51:57.231 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-29T15:51:57.233 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.267 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.042) 0:00:36.617 ******* 2020-11-29T15:51:57.273 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-29T15:51:57.275 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.310 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.042) 0:00:36.659 ******* 2020-11-29T15:51:57.316 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-29T15:51:57.317 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.354 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.044) 0:00:36.704 ******* 2020-11-29T15:51:57.360 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-29T15:51:57.362 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.396 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:36.745 ******* 2020-11-29T15:51:57.403 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-29T15:51:57.404 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.438 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:36.787 ******* 2020-11-29T15:51:57.444 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-29T15:51:57.445 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.480 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.042) 0:00:36.829 ******* 2020-11-29T15:51:57.485 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-29T15:51:57.487 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.521 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:36.870 ******* 2020-11-29T15:51:57.527 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-29T15:51:57.528 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.565 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.044) 0:00:36.914 ******* 2020-11-29T15:51:57.571 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-29T15:51:57.572 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.608 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.042) 0:00:36.957 ******* 2020-11-29T15:51:57.614 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-29T15:51:57.616 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.651 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.043) 0:00:37.000 ******* 2020-11-29T15:51:57.657 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-29T15:51:57.658 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.697 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.045) 0:00:37.046 ******* 2020-11-29T15:51:57.705 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-29T15:51:57.706 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.742 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.045) 0:00:37.091 ******* 2020-11-29T15:51:57.748 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-29T15:51:57.750 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.787 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.044) 0:00:37.136 ******* 2020-11-29T15:51:57.793 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-29T15:51:57.795 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.830 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.043) 0:00:37.179 ******* 2020-11-29T15:51:57.836 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-29T15:51:57.837 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.872 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:37.221 ******* 2020-11-29T15:51:57.878 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-29T15:51:57.879 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.913 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:37.262 ******* 2020-11-29T15:51:57.919 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-29T15:51:57.920 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.958 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.044) 0:00:37.307 ******* 2020-11-29T15:51:57.963 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-29T15:51:57.964 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:57.999 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:57 +0000 (0:00:00.041) 0:00:37.348 ******* 2020-11-29T15:51:58.025 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-29T15:51:58.027 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:58.060 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.061) 0:00:37.410 ******* 2020-11-29T15:51:58.088 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-29T15:51:58.090 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005208", "end": "2020-11-29 15:51:57.861724", "rc": 0, "start": "2020-11-29 15:51:57.856516", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:51:58.125 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.064) 0:00:37.474 ******* 2020-11-29T15:51:58.152 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-29T15:51:58.188 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.062) 0:00:37.537 ******* 2020-11-29T15:51:58.206 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-29T15:51:58.208 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005043", "end": "2020-11-29 15:51:57.776613", "rc": 0, "start": "2020-11-29 15:51:57.771570", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:51:58.262 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.074) 0:00:37.612 ******* 2020-11-29T15:51:58.290 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-29T15:51:58.341 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.078) 0:00:37.691 ******* 2020-11-29T15:51:58.486 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-29T15:51:58.488 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-29T15:51:58.532 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.190) 0:00:37.881 ******* 2020-11-29T15:51:58.560 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-29T15:51:58.562 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:58.597 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.064) 0:00:37.946 ******* 2020-11-29T15:51:58.655 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-29T15:51:58.657 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-11-29T15:51:58.693 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.095) 0:00:38.042 ******* 2020-11-29T15:51:58.725 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-29T15:51:58.727 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:51:58.764 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:51:58 +0000 (0:00:00.071) 0:00:38.113 ******* 2020-11-29T15:51:59.910 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-29T15:51:59.912 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-29T15:51:59.932 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-29T15:52:00.836 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-29T15:52:00.847 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-29T15:52:00.883 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:00 +0000 (0:00:02.118) 0:00:40.232 ******* 2020-11-29T15:52:00.916 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:00 +0000 (0:00:00.033) 0:00:40.265 ******* 2020-11-29T15:52:01.770 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-29T15:52:01.775 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:52:01.808 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:01 +0000 (0:00:00.891) 0:00:41.157 ******* 2020-11-29T15:52:01.836 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:52:01.838 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-29T15:52:01.872 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:01 +0000 (0:00:00.064) 0:00:41.222 ******* 2020-11-29T15:52:01.908 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:01 +0000 (0:00:00.035) 0:00:41.257 ******* 2020-11-29T15:52:01.973 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:01 +0000 (0:00:00.065) 0:00:41.322 ******* 2020-11-29T15:52:02.035 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-29T15:52:02.037 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033170", "end": "2020-11-29 15:52:01.797865", "rc": 0, "start": "2020-11-29 15:52:01.764695", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:02.071 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:02 +0000 (0:00:00.098) 0:00:41.421 ******* 2020-11-29T15:52:02.153 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.031839", "end": "2020-11-29 15:52:01.723405", "rc": 0, "start": "2020-11-29 15:52:01.691566", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:02.192 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:02 +0000 (0:00:00.120) 0:00:41.541 ******* 2020-11-29T15:52:04.941 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-29T15:52:04.943 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:52:04.977 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:04 +0000 (0:00:02.784) 0:00:44.326 ******* 2020-11-29T15:52:04.983 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:52:05.017 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:05 +0000 (0:00:00.040) 0:00:44.366 ******* 2020-11-29T15:52:14.087 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-29T15:52:14.089 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102076 files and directories 2020-11-29T15:52:14.090 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-11-29T15:52:14.092 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102076 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-29T15:52:14.123 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:14 +0000 (0:00:09.105) 0:00:53.472 ******* 2020-11-29T15:52:14.130 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102076 files and directories 2020-11-29T15:52:14.132 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-11-29T15:52:14.134 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102076 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-29T15:52:14.168 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:14 +0000 (0:00:00.044) 0:00:53.517 ******* 2020-11-29T15:52:14.989 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-29T15:52:14.991 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:52:15.026 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:15 +0000 (0:00:00.857) 0:00:54.375 ******* 2020-11-29T15:52:15.043 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:52:15.081 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:15 +0000 (0:00:00.054) 0:00:54.430 ******* 2020-11-29T15:52:15.087 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-29T15:52:15.088 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:15.135 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:15 +0000 (0:00:00.053) 0:00:54.484 ******* 2020-11-29T15:52:15.140 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:15.177 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:15 +0000 (0:00:00.042) 0:00:54.526 ******* 2020-11-29T15:52:23.866 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-29T15:52:23.867 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9898 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-11-29T15:52:23.869 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101934 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-11-29T15:52:23.870 INFO:teuthology.task.ansible.out:1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9898 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-11-29T15:52:23.872 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101934 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} changed: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow 2020-11-29T15:52:23.873 INFO:teuthology.task.ansible.out: instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9634 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading d 2020-11-29T15:52:23.875 INFO:teuthology.task.ansible.out:atabase ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101934 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additio 2020-11-29T15:52:23.877 INFO:teuthology.task.ansible.out:nal packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9634 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101934 files and directories currently installed.)", "Preparing to unp 2020-11-29T15:52:23.878 INFO:teuthology.task.ansible.out:ack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-29T15:52:23.906 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:23 +0000 (0:00:08.728) 0:01:03.255 ******* 2020-11-29T15:52:23.939 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:23 +0000 (0:00:00.033) 0:01:03.288 ******* 2020-11-29T15:52:24.785 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-29T15:52:24.787 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:52:24.821 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:24 +0000 (0:00:00.881) 0:01:04.170 ******* 2020-11-29T15:52:24.853 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:24 +0000 (0:00:00.032) 0:01:04.202 ******* 2020-11-29T15:52:24.942 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-29T15:52:24.944 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-29T15:52:24.980 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:24 +0000 (0:00:00.127) 0:01:04.329 ******* 2020-11-29T15:52:24.987 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-29T15:52:25.024 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.044) 0:01:04.373 ******* 2020-11-29T15:52:25.031 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-29T15:52:25.032 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:25.069 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.044) 0:01:04.418 ******* 2020-11-29T15:52:25.076 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:25.115 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.046) 0:01:04.464 ******* 2020-11-29T15:52:25.205 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-29T15:52:25.207 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-29T15:52:25.242 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.126) 0:01:04.591 ******* 2020-11-29T15:52:25.248 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-29T15:52:25.284 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.042) 0:01:04.633 ******* 2020-11-29T15:52:25.563 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-29T15:52:25.565 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-29T15:52:25.601 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.317) 0:01:04.950 ******* 2020-11-29T15:52:25.620 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-29T15:52:25.663 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.062) 0:01:05.012 ******* 2020-11-29T15:52:25.752 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-29T15:52:25.754 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1235, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-29T15:52:25.788 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.124) 0:01:05.137 ******* 2020-11-29T15:52:25.815 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1235, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-29T15:52:25.873 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.085) 0:01:05.222 ******* 2020-11-29T15:52:25.913 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-29T15:52:25.915 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-29T15:52:25.978 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:25 +0000 (0:00:00.105) 0:01:05.327 ******* 2020-11-29T15:52:26.012 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-29T15:52:26.049 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.070) 0:01:05.399 ******* 2020-11-29T15:52:26.287 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sun 2020-11-29 15:50:32 UTC", "ActiveEnterTimestampMonotonic": "19551082", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target systemd-tmpfiles-setup.service remote-fs.target systemd-journald.socket system.slice nss-lookup.target network.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2020-11-29 15:50:31 UTC", "AssertTimestampMonotonic": "19255958", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2020-11-29 15:50:31 UTC", "ConditionTimestampMonotonic": "19255957", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", 2020-11-29T15:52:26.289 INFO:teuthology.task.ansible.out:"Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1404", "ExecMainStartTimestamp": "Sun 2020-11-29 15:50:32 UTC", "ExecMainStartTimestampMonotonic": "19551071", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2020-11-29 15:50:31 UTC", "InactiveExitTimestampMonotonic": "19256660", "InvocationID": "9f64880206d649c788d4fa2440069e51", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA 2020-11-29T15:52:26.291 INFO:teuthology.task.ansible.out:": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1404", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", 2020-11-29T15:52:26.292 INFO:teuthology.task.ansible.out: "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2020-11-29 15:50:32 UTC", "StateChangeTimestampMonotonic": "19551082", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2020-11-29 15:50:32 UTC", "WatchdogTimestampMonotonic": "19551081", "WatchdogUSec": "0"}} 2020-11-29T15:52:26.324 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.274) 0:01:05.673 ******* 2020-11-29T15:52:26.351 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-29T15:52:26.353 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.387 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.062) 0:01:05.736 ******* 2020-11-29T15:52:26.394 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-29T15:52:26.395 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sun 2020-11-29 15:49:59 UTC", "ActiveEnterTimestampMonotonic": "19516554", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target systemd-journald.socket system.slice network.target systemd-tmpfiles-setup.service nss-lookup.target sysinit.target basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2020-11-29 15:49:58 UTC", "AssertTimestampMonotonic": "19279713", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2020-11-29 15:49:58 UTC", "ConditionTimestampMonotonic": "19279713", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-29T15:52:26.396 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1590", "ExecMainStartTimestamp": "Sun 2020-11-29 15:49:59 UTC", "ExecMainStartTimestampMonotonic": "19516542", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2020-11-29 15:49:58 UTC", "InactiveExitTimestampMonotonic": "19280616", "InvocationID": "247e5d5b2e69427594e2ff7e05cfcc2f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-29T15:52:26.398 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1590", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-29T15:52:26.400 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2020-11-29 15:49:59 UTC", "StateChangeTimestampMonotonic": "19516554", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2020-11-29 15:49:59 UTC", "WatchdogTimestampMonotonic": "19516552", "WatchdogUSec": "0"}} 2020-11-29T15:52:26.432 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.044) 0:01:05.781 ******* 2020-11-29T15:52:26.437 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-29T15:52:26.439 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.474 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.042) 0:01:05.823 ******* 2020-11-29T15:52:26.480 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-29T15:52:26.482 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.516 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.041) 0:01:05.865 ******* 2020-11-29T15:52:26.521 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-29T15:52:26.523 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.559 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.043) 0:01:05.908 ******* 2020-11-29T15:52:26.566 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-29T15:52:26.567 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.604 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.044) 0:01:05.953 ******* 2020-11-29T15:52:26.609 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-29T15:52:26.611 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.651 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.047) 0:01:06.001 ******* 2020-11-29T15:52:26.658 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-29T15:52:26.660 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.697 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.045) 0:01:06.046 ******* 2020-11-29T15:52:26.704 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-29T15:52:26.705 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.741 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.043) 0:01:06.090 ******* 2020-11-29T15:52:26.747 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-29T15:52:26.749 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.787 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.043) 0:01:06.133 ******* 2020-11-29T15:52:26.792 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-29T15:52:26.793 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.829 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.045) 0:01:06.179 ******* 2020-11-29T15:52:26.835 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-29T15:52:26.837 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.872 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.042) 0:01:06.221 ******* 2020-11-29T15:52:26.878 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-29T15:52:26.880 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.915 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.042) 0:01:06.264 ******* 2020-11-29T15:52:26.921 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-29T15:52:26.923 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:26.967 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:26 +0000 (0:00:00.051) 0:01:06.316 ******* 2020-11-29T15:52:26.973 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-29T15:52:26.975 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.010 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.043) 0:01:06.359 ******* 2020-11-29T15:52:27.020 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-29T15:52:27.022 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.062 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.051) 0:01:06.411 ******* 2020-11-29T15:52:27.068 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:27.070 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.109 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.047) 0:01:06.458 ******* 2020-11-29T15:52:27.115 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-29T15:52:27.117 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.156 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.047) 0:01:06.505 ******* 2020-11-29T15:52:27.162 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-29T15:52:27.164 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.203 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.046) 0:01:06.552 ******* 2020-11-29T15:52:27.209 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:27.211 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.259 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.056) 0:01:06.608 ******* 2020-11-29T15:52:27.267 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-29T15:52:27.269 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.314 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.055) 0:01:06.664 ******* 2020-11-29T15:52:27.321 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-29T15:52:27.323 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.358 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.043) 0:01:06.707 ******* 2020-11-29T15:52:27.365 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-29T15:52:27.367 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.404 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.045) 0:01:06.753 ******* 2020-11-29T15:52:27.410 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-29T15:52:27.412 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.471 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.066) 0:01:06.820 ******* 2020-11-29T15:52:27.477 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-29T15:52:27.479 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.514 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.043) 0:01:06.863 ******* 2020-11-29T15:52:27.521 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-29T15:52:27.522 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.573 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.058) 0:01:06.922 ******* 2020-11-29T15:52:27.579 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-29T15:52:27.581 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.616 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.042) 0:01:06.965 ******* 2020-11-29T15:52:27.622 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-29T15:52:27.623 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.658 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.042) 0:01:07.007 ******* 2020-11-29T15:52:27.664 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-29T15:52:27.666 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.701 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.042) 0:01:07.050 ******* 2020-11-29T15:52:27.710 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.747 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.046) 0:01:07.096 ******* 2020-11-29T15:52:27.753 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-29T15:52:27.754 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.797 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.049) 0:01:07.146 ******* 2020-11-29T15:52:27.803 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-29T15:52:27.805 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.862 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.065) 0:01:07.211 ******* 2020-11-29T15:52:27.871 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-29T15:52:27.873 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.931 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.069) 0:01:07.281 ******* 2020-11-29T15:52:27.940 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-29T15:52:27.942 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:27.989 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:27 +0000 (0:00:00.057) 0:01:07.338 ******* 2020-11-29T15:52:27.994 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-29T15:52:27.996 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.035 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.046) 0:01:07.384 ******* 2020-11-29T15:52:28.043 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-29T15:52:28.044 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.081 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.045) 0:01:07.430 ******* 2020-11-29T15:52:28.088 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-29T15:52:28.090 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.126 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.044) 0:01:07.475 ******* 2020-11-29T15:52:28.134 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-29T15:52:28.135 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.174 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.047) 0:01:07.523 ******* 2020-11-29T15:52:28.180 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-29T15:52:28.182 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.218 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.044) 0:01:07.567 ******* 2020-11-29T15:52:28.224 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-29T15:52:28.226 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.264 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.046) 0:01:07.613 ******* 2020-11-29T15:52:28.274 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.309 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.045) 0:01:07.659 ******* 2020-11-29T15:52:28.315 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-29T15:52:28.317 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.353 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.043) 0:01:07.702 ******* 2020-11-29T15:52:28.359 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-29T15:52:28.360 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.396 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.042) 0:01:07.745 ******* 2020-11-29T15:52:28.403 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-29T15:52:28.404 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.441 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.043) 0:01:07.788 ******* 2020-11-29T15:52:28.445 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-29T15:52:28.447 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.482 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.043) 0:01:07.832 ******* 2020-11-29T15:52:28.488 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-29T15:52:28.489 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.526 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.043) 0:01:07.875 ******* 2020-11-29T15:52:28.532 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-29T15:52:28.534 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.569 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.042) 0:01:07.918 ******* 2020-11-29T15:52:28.575 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-29T15:52:28.576 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.611 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.041) 0:01:07.960 ******* 2020-11-29T15:52:28.616 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-29T15:52:28.618 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.657 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.046) 0:01:08.006 ******* 2020-11-29T15:52:28.664 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-29T15:52:28.666 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.700 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.043) 0:01:08.049 ******* 2020-11-29T15:52:28.707 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-29T15:52:28.709 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.742 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.041) 0:01:08.091 ******* 2020-11-29T15:52:28.748 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-29T15:52:28.750 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.804 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.061) 0:01:08.153 ******* 2020-11-29T15:52:28.832 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-29T15:52:28.834 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:28.868 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.063) 0:01:08.217 ******* 2020-11-29T15:52:28.873 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-29T15:52:28.875 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-29T15:52:28.912 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:28 +0000 (0:00:00.044) 0:01:08.261 ******* 2020-11-29T15:52:28.992 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-11-29T15:52:29.034 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:29 +0000 (0:00:00.122) 0:01:08.383 ******* 2020-11-29T15:52:29.220 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-29T15:52:29.222 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-29T15:52:29.257 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:29 +0000 (0:00:00.222) 0:01:08.606 ******* 2020-11-29T15:52:29.284 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-29T15:52:29.285 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:29.319 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:29 +0000 (0:00:00.061) 0:01:08.668 ******* 2020-11-29T15:52:29.356 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-29T15:52:29.357 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-29T15:52:29.395 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:29 +0000 (0:00:00.076) 0:01:08.744 ******* 2020-11-29T15:52:29.422 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-29T15:52:29.424 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:29.460 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:29 +0000 (0:00:00.064) 0:01:08.809 ******* 2020-11-29T15:52:29.622 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-29T15:52:29.624 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-29T15:52:29.679 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:29 +0000 (0:00:00.218) 0:01:09.028 ******* 2020-11-29T15:52:29.790 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-29T15:52:29.830 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:29 +0000 (0:00:00.150) 0:01:09.179 ******* 2020-11-29T15:52:29.992 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-29T15:52:29.994 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-29T15:52:30.030 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.200) 0:01:09.379 ******* 2020-11-29T15:52:30.129 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-29T15:52:30.166 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.136) 0:01:09.516 ******* 2020-11-29T15:52:30.329 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-29T15:52:30.331 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-29T15:52:30.366 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.199) 0:01:09.715 ******* 2020-11-29T15:52:30.466 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-29T15:52:30.502 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.136) 0:01:09.852 ******* 2020-11-29T15:52:30.509 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-29T15:52:30.511 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-29T15:52:30.546 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.043) 0:01:09.896 ******* 2020-11-29T15:52:30.573 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-29T15:52:30.575 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:30.611 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.064) 0:01:09.961 ******* 2020-11-29T15:52:30.629 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-29T15:52:30.631 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-29T15:52:30.667 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.055) 0:01:10.016 ******* 2020-11-29T15:52:30.673 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-29T15:52:30.675 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:30.714 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.047) 0:01:10.063 ******* 2020-11-29T15:52:30.720 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-29T15:52:30.721 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:30.757 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.042) 0:01:10.106 ******* 2020-11-29T15:52:30.764 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-29T15:52:30.765 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:30.799 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.042) 0:01:10.148 ******* 2020-11-29T15:52:30.805 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-29T15:52:30.807 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:30.857 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.058) 0:01:10.207 ******* 2020-11-29T15:52:30.864 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-29T15:52:30.865 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:30.904 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.046) 0:01:10.253 ******* 2020-11-29T15:52:30.911 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-29T15:52:30.913 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:30.947 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.043) 0:01:10.297 ******* 2020-11-29T15:52:30.956 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-29T15:52:30.958 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:30.994 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:30 +0000 (0:00:00.046) 0:01:10.343 ******* 2020-11-29T15:52:31.000 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-29T15:52:31.002 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.037 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.043) 0:01:10.387 ******* 2020-11-29T15:52:31.043 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:31.045 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-29T15:52:31.081 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.043) 0:01:10.430 ******* 2020-11-29T15:52:31.087 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.125 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.044) 0:01:10.474 ******* 2020-11-29T15:52:31.131 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-29T15:52:31.133 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.143 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.145 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.154 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.156 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi168.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.191 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.065) 0:01:10.540 ******* 2020-11-29T15:52:31.197 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-29T15:52:31.200 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.200 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.211 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.232 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.234 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.236 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.280 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.089) 0:01:10.629 ******* 2020-11-29T15:52:31.287 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.287 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.288 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.288 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.288 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-29T15:52:31.302 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.304 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.313 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.316 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.324 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.326 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.336 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.347 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.391 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.111) 0:01:10.740 ******* 2020-11-29T15:52:31.491 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-29T15:52:31.493 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.009296", "end": "2020-11-29 15:52:31.463986", "rc": 0, "start": "2020-11-29 15:52:31.454690", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-29T15:52:31.551 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.159) 0:01:10.900 ******* 2020-11-29T15:52:31.581 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-29T15:52:31.583 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.624 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.073) 0:01:10.973 ******* 2020-11-29T15:52:31.631 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-29T15:52:31.633 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.007442", "end": "2020-11-29 15:52:31.578429", "rc": 0, "start": "2020-11-29 15:52:31.570987", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-29T15:52:31.679 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.055) 0:01:11.028 ******* 2020-11-29T15:52:31.685 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-29T15:52:31.687 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.730 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.050) 0:01:11.079 ******* 2020-11-29T15:52:31.736 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-29T15:52:31.737 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.777 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.047) 0:01:11.126 ******* 2020-11-29T15:52:31.783 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-29T15:52:31.784 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.823 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.045) 0:01:11.172 ******* 2020-11-29T15:52:31.829 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-29T15:52:31.830 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.867 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.043) 0:01:11.216 ******* 2020-11-29T15:52:31.893 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-29T15:52:31.894 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:31.929 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:31 +0000 (0:00:00.062) 0:01:11.278 ******* 2020-11-29T15:52:32.743 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:52:32.780 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:32 +0000 (0:00:00.850) 0:01:12.129 ******* 2020-11-29T15:52:32.806 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-29T15:52:32.808 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:32.841 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:32 +0000 (0:00:00.061) 0:01:12.191 ******* 2020-11-29T15:52:32.848 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-29T15:52:32.849 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:52:32.885 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:32 +0000 (0:00:00.043) 0:01:12.235 ******* 2020-11-29T15:52:32.891 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-29T15:52:32.892 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item={'block_used': 13646002, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449106, 'size_available': 927535538176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113954, 'block_size': 4096, 'inode_available': 60941022}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449106, "block_size": 4096, "block_total": 240095108, "block_used": 13646002, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941022, "inode_total": 61054976, "inode_used": 113954, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927535538176, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}} 2020-11-29T15:52:32.939 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:32 +0000 (0:00:00.053) 0:01:12.288 ******* 2020-11-29T15:52:32.946 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-29T15:52:32.947 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:32.989 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:32 +0000 (0:00:00.050) 0:01:12.338 ******* 2020-11-29T15:52:33.041 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi184.front.sepia.ceph.com] => (item={'block_used': 13646020, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449088, 'size_available': 927535464448, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113954, 'block_size': 4096, 'inode_available': 60941022}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449088, "block_size": 4096, "block_total": 240095108, "block_used": 13646020, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941022, "inode_total": 61054976, "inode_used": 113954, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927535464448, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}} 2020-11-29T15:52:33.076 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:33 +0000 (0:00:00.086) 0:01:12.425 ******* 2020-11-29T15:52:33.082 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-29T15:52:33.083 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010902", "end": "2020-11-29 15:52:32.870559", "rc": 0, "start": "2020-11-29 15:52:32.859657", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:33.125 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:33 +0000 (0:00:00.048) 0:01:12.474 ******* 2020-11-29T15:52:33.166 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-29T15:52:33.168 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'block_used': 13646002, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449106, 'size_available': 927535538176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113954, 'block_size': 4096, 'inode_available': 60941022}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449106, "block_size": 4096, "block_total": 240095108, "block_used": 13646002, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941022, "inode_total": 61054976, "inode_used": 113954, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927535538176, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:33.214 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:33 +0000 (0:00:00.082) 0:01:12.557 ******* TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-29T15:52:33.216 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010916", "end": "2020-11-29 15:52:32.807948", "rc": 0, "start": "2020-11-29 15:52:32.797032", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:33.252 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:33 +0000 (0:00:00.044) 0:01:12.601 ******* 2020-11-29T15:52:33.258 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-29T15:52:33.296 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:33 +0000 (0:00:00.043) 0:01:12.645 ******* 2020-11-29T15:52:33.302 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-29T15:52:33.303 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'block_used': 13646020, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449088, 'size_available': 927535464448, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113954, 'block_size': 4096, 'inode_available': 60941022}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449088, "block_size": 4096, "block_total": 240095108, "block_used": 13646020, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941022, "inode_total": 61054976, "inode_used": 113954, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927535464448, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:33.343 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:33 +0000 (0:00:00.047) 0:01:12.692 ******* 2020-11-29T15:52:33.351 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-29T15:52:33.352 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:33.362 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-29T15:52:33.400 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:33 +0000 (0:00:00.056) 0:01:12.749 ******* 2020-11-29T15:52:33.440 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-29T15:52:33.442 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:35.474 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqZ4W5wM7aZmiSU63lFbGvqzjykwjPHKdl'], 'uuids': ['2e514a92-8889-4ad5-896b-02d0f7c0559b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.909643", "end": "2020-11-29 15:52:35.062732", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqZ4W5wM7aZmiSU63lFbGvqzjykwjPHKdl"], "labels": [], "masters": [], "uuids": ["2e514a92-8889-4ad5-896b-02d0f7c0559b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:33.153089", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:35.486 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:35.497 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:35.562 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkYqLvaMPZBkfqGjgPOFXowEt9kX2tzuuu'], 'uuids': ['273b6784-6cf9-45aa-bf01-19c729c37963']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.093674", "end": "2020-11-29 15:52:35.335589", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkYqLvaMPZBkfqGjgPOFXowEt9kX2tzuuu"], "labels": [], "masters": [], "uuids": ["273b6784-6cf9-45aa-bf01-19c729c37963"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:33.241915", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:35.563 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:35.575 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:36.720 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqNCWIduiwDP1ZioAwERF8dymztNgjYimM'], 'uuids': ['ab1af0bc-cca9-4de8-87f0-e450a0452f8e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.109494", "end": "2020-11-29 15:52:36.318322", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqNCWIduiwDP1ZioAwERF8dymztNgjYimM"], "labels": [], "masters": [], "uuids": ["ab1af0bc-cca9-4de8-87f0-e450a0452f8e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:35.208828", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:36.732 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLJY', 'wwn-0x5000c5009294076e'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLJY-part1', 'wwn-0x5000c5009294076e-part1'], 'uuids': ['d8c38c2f-ff33-4bf4-853e-25622c47cb18']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009294076e', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XLJY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLJY", "wwn-0x5000c5009294076e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLJY-part1", "wwn-0x5000c5009294076e-part1"], "labels": [], "masters": [], "uuids": ["d8c38c2f-ff33-4bf4-853e-25622c47cb18"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLJY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294076e"}}, "skip_reason": 2020-11-29T15:52:36.734 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-29T15:52:36.819 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkE2bv9XuKRoGVjkkoY5M3CrmgtE4Gqcas'], 'uuids': ['9cca943b-89b4-470f-af4f-009a8c6764c6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.113150", "end": "2020-11-29 15:52:36.593219", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkE2bv9XuKRoGVjkkoY5M3CrmgtE4Gqcas"], "labels": [], "masters": [], "uuids": ["9cca943b-89b4-470f-af4f-009a8c6764c6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:35.480069", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:36.821 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFGH', 'wwn-0x5000c50092924459'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFGH-part1', 'wwn-0x5000c50092924459-part1'], 'uuids': ['d8c38c2f-ff33-4bf4-853e-25622c47cb18']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092924459', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFGH', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFGH", "wwn-0x5000c50092924459"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFGH-part1", "wwn-0x5000c50092924459-part1"], "labels": [], "masters": [], "uuids": ["d8c38c2f-ff33-4bf4-853e-25622c47cb18"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFGH", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924459"}}, "skip_reason": 2020-11-29T15:52:36.822 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-29T15:52:37.980 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-t0w4dS-SCYy-cPrg-wFHH-Z2Wh-9CHC-G19ZJl', 'nvme-INTEL_SSDPEDMD400G4_CVFT626500CE400BGN', 'nvme-nvme.8086-43564654363236353030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.128274", "end": "2020-11-29 15:52:37.572790", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-t0w4dS-SCYy-cPrg-wFHH-Z2Wh-9CHC-G19ZJl", "nvme-INTEL_SSDPEDMD400G4_CVFT626500CE400BGN", "nvme-nvme.8086-43564654363236353030434534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:36.444516", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-29T15:52:37.981 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:37.991 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:37.992 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:38.088 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ZeWoQQ-6Ry2-Pp4Z-QFo2-Fb5X-29bs-bwqkgy', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300KZ400BGN', 'nvme-nvme.8086-435646543632333330304b5a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.137944", "end": "2020-11-29 15:52:37.865128", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ZeWoQQ-6Ry2-Pp4Z-QFo2-Fb5X-29bs-bwqkgy", "nvme-INTEL_SSDPEDMD400G4_CVFT623300KZ400BGN", "nvme-nvme.8086-435646543632333330304b5a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:36.727184", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-29T15:52:38.089 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:38.100 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:39.254 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqcmBUQMiy1u19yNtlVa1JqSrwjP4zWok4'], 'uuids': ['31693ec3-6755-4480-b414-a19f7a5b9bc0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.140766", "end": "2020-11-29 15:52:38.845398", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqcmBUQMiy1u19yNtlVa1JqSrwjP4zWok4"], "labels": [], "masters": [], "uuids": ["31693ec3-6755-4480-b414-a19f7a5b9bc0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:37.704632", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:39.267 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:39.269 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:39.502 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkQmtq2sCZzCz1w5TyZ93rQz8EAqWiSduv'], 'uuids': ['6e8bb336-c929-4988-b9a2-89d37a48d0e3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.266303", "end": "2020-11-29 15:52:39.285746", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkQmtq2sCZzCz1w5TyZ93rQz8EAqWiSduv"], "labels": [], "masters": [], "uuids": ["6e8bb336-c929-4988-b9a2-89d37a48d0e3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:38.019443", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:39.513 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:39.525 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:40.533 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqzBM3OzYiczofxb9PPxbvPb3wHJABw0U8'], 'uuids': ['3072f972-aab1-4911-a3ba-6b912a4d6a7c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.136537", "end": "2020-11-29 15:52:40.124426", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqzBM3OzYiczofxb9PPxbvPb3wHJABw0U8"], "labels": [], "masters": [], "uuids": ["3072f972-aab1-4911-a3ba-6b912a4d6a7c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:38.987889", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:40.544 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:40.765 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetk8In6QCrs6c1Zyo73qJn1AJO0pnnHtaNT'], 'uuids': ['f672927b-076d-4db6-b000-e80ba93c17a4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.120678", "end": "2020-11-29 15:52:40.545908", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetk8In6QCrs6c1Zyo73qJn1AJO0pnnHtaNT"], "labels": [], "masters": [], "uuids": ["f672927b-076d-4db6-b000-e80ba93c17a4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:39.425230", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:40.777 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:41.785 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqY22GXjSTAOpNVsWpo13R5aC8uyI3iqH4'], 'uuids': ['1de30062-b896-4063-8afe-a63c51cae356']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.117559", "end": "2020-11-29 15:52:41.372666", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqY22GXjSTAOpNVsWpo13R5aC8uyI3iqH4"], "labels": [], "masters": [], "uuids": ["1de30062-b896-4063-8afe-a63c51cae356"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:40.255107", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:41.835 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:41 +0000 (0:00:08.435) 0:01:21.185 ******* 2020-11-29T15:52:41.887 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-29T15:52:41.887 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.160 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqZ4W5wM7aZmiSU63lFbGvqzjykwjPHKdl'], 'uuids': ['2e514a92-8889-4ad5-896b-02d0f7c0559b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.155975", "end": "2020-11-29 15:52:41.757355", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqZ4W5wM7aZmiSU63lFbGvqzjykwjPHKdl"], "labels": [], "masters": [], "uuids": ["2e514a92-8889-4ad5-896b-02d0f7c0559b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:41.601380", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15289 s, 754 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15289 s, 754 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:42.172 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.184 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.227 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-29T15:52:42.229 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetk2eKQ9Vl9SILUsl5SDgH2VTiV82ZbuZNh'], 'uuids': ['10322985-03ad-4c63-aaaa-4287090c397d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.321086", "end": "2020-11-29 15:52:42.005490", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetk2eKQ9Vl9SILUsl5SDgH2VTiV82ZbuZNh"], "labels": [], "masters": [], "uuids": ["10322985-03ad-4c63-aaaa-4287090c397d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:40.684404", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-29T15:52:42.273 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:42 +0000 (0:00:00.438) 0:01:21.623 ******* 2020-11-29T15:52:42.332 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-29T15:52:42.334 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.479 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqNCWIduiwDP1ZioAwERF8dymztNgjYimM'], 'uuids': ['ab1af0bc-cca9-4de8-87f0-e450a0452f8e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.166061", "end": "2020-11-29 15:52:42.069168", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqNCWIduiwDP1ZioAwERF8dymztNgjYimM"], "labels": [], "masters": [], "uuids": ["ab1af0bc-cca9-4de8-87f0-e450a0452f8e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:41.903107", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.16137 s, 715 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.16137 s, 715 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:42.480 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLJY', 'wwn-0x5000c5009294076e'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLJY-part1', 'wwn-0x5000c5009294076e-part1'], 'uuids': ['d8c38c2f-ff33-4bf4-853e-25622c47cb18']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009294076e', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XLJY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLJY", "wwn-0x5000c5009294076e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLJY-part1", "wwn-0x5000c5009294076e-part1"], "labels": [], "masters": [], "uuids": ["d8c38c2f-ff33-4bf4-853e-25622c47cb18"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLJY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294076e"}}, "skip_reason": 2020-11-29T15:52:42.482 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-29T15:52:42.629 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkYqLvaMPZBkfqGjgPOFXowEt9kX2tzuuu'], 'uuids': ['273b6784-6cf9-45aa-bf01-19c729c37963']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.168054", "end": "2020-11-29 15:52:42.413035", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkYqLvaMPZBkfqGjgPOFXowEt9kX2tzuuu"], "labels": [], "masters": [], "uuids": ["273b6784-6cf9-45aa-bf01-19c729c37963"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:42.244981", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.163429 s, 706 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.163429 s, 706 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:42.652 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.655 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-t0w4dS-SCYy-cPrg-wFHH-Z2Wh-9CHC-G19ZJl', 'nvme-INTEL_SSDPEDMD400G4_CVFT626500CE400BGN', 'nvme-nvme.8086-43564654363236353030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.053691", "end": "2020-11-29 15:52:42.248911", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-t0w4dS-SCYy-cPrg-wFHH-Z2Wh-9CHC-G19ZJl", "nvme-INTEL_SSDPEDMD400G4_CVFT626500CE400BGN", "nvme-nvme.8086-43564654363236353030434534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:42.195220", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0490192 s, 2.4 GB/s", "std 2020-11-29T15:52:42.657 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0490192 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:42.668 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.669 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.682 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.955 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkE2bv9XuKRoGVjkkoY5M3CrmgtE4Gqcas'], 'uuids': ['9cca943b-89b4-470f-af4f-009a8c6764c6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.161630", "end": "2020-11-29 15:52:42.739168", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkE2bv9XuKRoGVjkkoY5M3CrmgtE4Gqcas"], "labels": [], "masters": [], "uuids": ["9cca943b-89b4-470f-af4f-009a8c6764c6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:42.577538", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.156811 s, 736 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.156811 s, 736 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:42.957 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqcmBUQMiy1u19yNtlVa1JqSrwjP4zWok4'], 'uuids': ['31693ec3-6755-4480-b414-a19f7a5b9bc0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.148864", "end": "2020-11-29 15:52:42.550377", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqcmBUQMiy1u19yNtlVa1JqSrwjP4zWok4"], "labels": [], "masters": [], "uuids": ["31693ec3-6755-4480-b414-a19f7a5b9bc0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:42.401513", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146141 s, 789 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146141 s, 789 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:42.969 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFGH', 'wwn-0x5000c50092924459'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFGH-part1', 'wwn-0x5000c50092924459-part1'], 'uuids': ['d8c38c2f-ff33-4bf4-853e-25622c47cb18']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092924459', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFGH', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFGH", "wwn-0x5000c50092924459"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFGH-part1", "wwn-0x5000c50092924459-part1"], "labels": [], "masters": [], "uuids": ["d8c38c2f-ff33-4bf4-853e-25622c47cb18"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFGH", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092924459"}}, "skip_reason": 2020-11-29T15:52:42.971 INFO:teuthology.task.ansible.out:"Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:42.982 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:43.129 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ZeWoQQ-6Ry2-Pp4Z-QFo2-Fb5X-29bs-bwqkgy', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300KZ400BGN', 'nvme-nvme.8086-435646543632333330304b5a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.053564", "end": "2020-11-29 15:52:42.917392", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ZeWoQQ-6Ry2-Pp4Z-QFo2-Fb5X-29bs-bwqkgy", "nvme-INTEL_SSDPEDMD400G4_CVFT623300KZ400BGN", "nvme-nvme.8086-435646543632333330304b5a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:42.863828", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0487974 s, 2.4 GB/s", "std 2020-11-29T15:52:43.131 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0487974 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:43.141 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:43.152 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:43.258 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqzBM3OzYiczofxb9PPxbvPb3wHJABw0U8'], 'uuids': ['3072f972-aab1-4911-a3ba-6b912a4d6a7c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.159869", "end": "2020-11-29 15:52:42.850056", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqzBM3OzYiczofxb9PPxbvPb3wHJABw0U8"], "labels": [], "masters": [], "uuids": ["3072f972-aab1-4911-a3ba-6b912a4d6a7c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:42.690187", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.156792 s, 736 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.156792 s, 736 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:43.269 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:43.427 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkQmtq2sCZzCz1w5TyZ93rQz8EAqWiSduv'], 'uuids': ['6e8bb336-c929-4988-b9a2-89d37a48d0e3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.163097", "end": "2020-11-29 15:52:43.215074", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetkQmtq2sCZzCz1w5TyZ93rQz8EAqWiSduv"], "labels": [], "masters": [], "uuids": ["6e8bb336-c929-4988-b9a2-89d37a48d0e3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:43.051977", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159619 s, 723 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159619 s, 723 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:43.438 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:43.460 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:43.546 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqY22GXjSTAOpNVsWpo13R5aC8uyI3iqH4'], 'uuids': ['1de30062-b896-4063-8afe-a63c51cae356']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.155114", "end": "2020-11-29 15:52:43.139389", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-HeVwrozMYfFSS2QUhh4XPr1Hz6Uz3OVqY22GXjSTAOpNVsWpo13R5aC8uyI3iqH4"], "labels": [], "masters": [], "uuids": ["1de30062-b896-4063-8afe-a63c51cae356"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:42.984275", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150363 s, 767 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150363 s, 767 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:43.607 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:43 +0000 (0:00:01.333) 0:01:22.956 ******* 2020-11-29T15:52:43.741 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetk8In6QCrs6c1Zyo73qJn1AJO0pnnHtaNT'], 'uuids': ['f672927b-076d-4db6-b000-e80ba93c17a4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.162995", "end": "2020-11-29 15:52:43.527422", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetk8In6QCrs6c1Zyo73qJn1AJO0pnnHtaNT"], "labels": [], "masters": [], "uuids": ["f672927b-076d-4db6-b000-e80ba93c17a4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:43.364427", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158212 s, 729 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158212 s, 729 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:43.753 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:43.869 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-29T15:52:43.871 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.133113", "end": "2020-11-29 15:52:43.464531", "rc": 0, "start": "2020-11-29 15:52:43.331418", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:43.915 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:43 +0000 (0:00:00.307) 0:01:23.264 ******* 2020-11-29T15:52:44.025 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-29T15:52:44.027 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetk2eKQ9Vl9SILUsl5SDgH2VTiV82ZbuZNh'], 'uuids': ['10322985-03ad-4c63-aaaa-4287090c397d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.159421", "end": "2020-11-29 15:52:43.810393", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TxWkNljwDGYCBofFW38I6cVRukVaZetk2eKQ9Vl9SILUsl5SDgH2VTiV82ZbuZNh"], "labels": [], "masters": [], "uuids": ["10322985-03ad-4c63-aaaa-4287090c397d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-29 15:52:43.650972", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154697 s, 746 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154697 s, 746 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:44.084 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.169) 0:01:23.433 ******* 2020-11-29T15:52:44.104 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-29T15:52:44.105 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.056296", "end": "2020-11-29 15:52:43.696961", "rc": 0, "start": "2020-11-29 15:52:43.640665", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:44.146 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.061) 0:01:23.495 ******* 2020-11-29T15:52:44.173 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-29T15:52:44.211 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.065) 0:01:23.560 ******* 2020-11-29T15:52:44.237 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-29T15:52:44.239 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.275 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.064) 0:01:23.624 ******* 2020-11-29T15:52:44.301 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-29T15:52:44.303 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.338 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.062) 0:01:23.687 ******* 2020-11-29T15:52:44.344 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-29T15:52:44.346 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.120142", "end": "2020-11-29 15:52:44.125067", "rc": 0, "start": "2020-11-29 15:52:44.004925", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:44.383 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.045) 0:01:23.732 ******* 2020-11-29T15:52:44.390 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.433 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.050) 0:01:23.783 ******* 2020-11-29T15:52:44.460 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-29T15:52:44.462 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.497 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.063) 0:01:23.846 ******* 2020-11-29T15:52:44.618 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-29T15:52:44.620 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Check for physical volumes] *********************************** changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.103069", "end": "2020-11-29 15:52:44.411795", "rc": 0, "start": "2020-11-29 15:52:44.308726", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:44.658 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.161) 0:01:24.007 ******* 2020-11-29T15:52:44.693 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.035) 0:01:24.043 ******* 2020-11-29T15:52:44.701 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-29T15:52:44.743 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.049) 0:01:24.092 ******* 2020-11-29T15:52:44.749 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-29T15:52:44.751 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'block_used': 13646020, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449088, 'size_available': 927535464448, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113954, 'block_size': 4096, 'inode_available': 60941022}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449088, "block_size": 4096, "block_total": 240095108, "block_used": 13646020, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941022, "inode_total": 61054976, "inode_used": 113954, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927535464448, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.787 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.044) 0:01:24.136 ******* 2020-11-29T15:52:44.793 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-29T15:52:44.795 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.832 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.044) 0:01:24.181 ******* 2020-11-29T15:52:44.838 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.875 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.042) 0:01:24.224 ******* 2020-11-29T15:52:44.881 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-29T15:52:44.883 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.920 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.045) 0:01:24.270 ******* 2020-11-29T15:52:44.926 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:44.928 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:44.963 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:44 +0000 (0:00:00.042) 0:01:24.312 ******* 2020-11-29T15:52:44.970 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-29T15:52:44.972 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.011 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.047) 0:01:24.360 ******* 2020-11-29T15:52:45.017 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:45.019 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.058 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.048) 0:01:24.408 ******* 2020-11-29T15:52:45.065 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-29T15:52:45.066 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.104 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.045) 0:01:24.453 ******* 2020-11-29T15:52:45.117 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-29T15:52:45.118 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.178 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.073) 0:01:24.527 ******* 2020-11-29T15:52:45.204 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-29T15:52:45.206 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.246 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.068) 0:01:24.595 ******* 2020-11-29T15:52:45.255 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-29T15:52:45.256 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-29T15:52:45.294 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.047) 0:01:24.643 ******* 2020-11-29T15:52:45.301 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-29T15:52:45.303 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.341 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.047) 0:01:24.690 ******* 2020-11-29T15:52:45.346 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-29T15:52:45.348 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'block_used': 13646002, 'uuid': 'd8c38c2f-ff33-4bf4-853e-25622c47cb18', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226449106, 'size_available': 927535538176, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113954, 'block_size': 4096, 'inode_available': 60941022}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226449106, "block_size": 4096, "block_total": 240095108, "block_used": 13646002, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941022, "inode_total": 61054976, "inode_used": 113954, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927535538176, "size_total": 983429562368, "uuid": "d8c38c2f-ff33-4bf4-853e-25622c47cb18"}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.385 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.043) 0:01:24.734 ******* 2020-11-29T15:52:45.412 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.448 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.062) 0:01:24.797 ******* 2020-11-29T15:52:45.474 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:45.476 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.512 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.064) 0:01:24.861 ******* 2020-11-29T15:52:45.538 INFO:teuthology.task.ansible.out: 2020-11-29T15:52:45.539 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:45.541 INFO:teuthology.task.ansible.out: skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.575 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.063) 0:01:24.924 ******* 2020-11-29T15:52:45.604 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-29T15:52:45.606 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.643 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.067) 0:01:24.992 ******* 2020-11-29T15:52:45.645 INFO:teuthology.task.ansible.out: 2020-11-29T15:52:45.671 INFO:teuthology.task.ansible.out: 2020-11-29T15:52:45.673 INFO:teuthology.task.ansible.out:TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.707 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.064) 0:01:25.056 ******* 2020-11-29T15:52:45.736 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:45.771 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.064) 0:01:25.121 ******* 2020-11-29T15:52:45.936 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-29T15:52:45.938 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-29T15:52:45.977 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:45 +0000 (0:00:00.205) 0:01:25.326 ******* 2020-11-29T15:52:46.142 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-29T15:52:46.181 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:46 +0000 (0:00:00.204) 0:01:25.530 ******* 2020-11-29T15:52:46.516 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:46.518 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:46.749 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:46.751 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:46.981 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:47.003 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:47.222 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:47.233 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:47.448 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:47.485 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:47 +0000 (0:00:01.304) 0:01:26.834 ******* 2020-11-29T15:52:47.492 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-29T15:52:47.534 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:47 +0000 (0:00:00.045) 0:01:26.880 ******* 2020-11-29T15:52:47.611 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-29T15:52:47.613 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-29T15:52:47.655 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:47 +0000 (0:00:00.124) 0:01:27.004 ******* 2020-11-29T15:52:47.672 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-29T15:52:47.711 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:47 +0000 (0:00:00.055) 0:01:27.060 ******* 2020-11-29T15:52:47.803 INFO:teuthology.task.ansible.out: 2020-11-29T15:52:47.805 INFO:teuthology.task.ansible.out:TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-29T15:52:47.859 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-29T15:52:47.912 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-29T15:52:47.966 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-29T15:52:48.029 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-29T15:52:48.083 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-29T15:52:48.136 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-29T15:52:48.148 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:48.188 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:48 +0000 (0:00:00.476) 0:01:27.537 ******* 2020-11-29T15:52:48.206 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-29T15:52:48.218 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:48.260 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:48 +0000 (0:00:00.072) 0:01:27.610 ******* 2020-11-29T15:52:48.312 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-29T15:52:48.350 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:48 +0000 (0:00:00.089) 0:01:27.699 ******* 2020-11-29T15:52:48.393 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-11-29T15:52:48.428 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:48 +0000 (0:00:00.077) 0:01:27.777 ******* 2020-11-29T15:52:48.461 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:48 +0000 (0:00:00.033) 0:01:27.810 ******* 2020-11-29T15:52:48.498 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-11-29T15:52:48.500 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-11-29T15:52:48.533 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:48 +0000 (0:00:00.072) 0:01:27.883 ******* 2020-11-29T15:52:48.929 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-29T15:52:48.930 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true} 2020-11-29T15:52:48.966 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:48 +0000 (0:00:00.432) 0:01:28.315 ******* 2020-11-29T15:52:48.972 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true} 2020-11-29T15:52:49.012 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:49 +0000 (0:00:00.046) 0:01:28.362 ******* 2020-11-29T15:52:49.388 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-29T15:52:49.389 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-29T15:52:49.426 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:49 +0000 (0:00:00.413) 0:01:28.775 ******* 2020-11-29T15:52:49.486 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-29T15:52:49.526 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:49 +0000 (0:00:00.099) 0:01:28.875 ******* 2020-11-29T15:52:49.565 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-29T15:52:49.568 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-29T15:52:49.602 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:49 +0000 (0:00:00.076) 0:01:28.951 ******* 2020-11-29T15:52:49.662 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-29T15:52:49.700 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:49 +0000 (0:00:00.097) 0:01:29.049 ******* 2020-11-29T15:52:49.738 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-29T15:52:49.740 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-29T15:52:49.777 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:49 +0000 (0:00:00.077) 0:01:29.126 ******* 2020-11-29T15:52:49.835 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-29T15:52:49.872 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:49 +0000 (0:00:00.094) 0:01:29.221 ******* 2020-11-29T15:52:50.098 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-29T15:52:50.100 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1235, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-29T15:52:50.136 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:50 +0000 (0:00:00.263) 0:01:29.485 ******* 2020-11-29T15:52:50.185 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1235, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-29T15:52:50.229 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:50 +0000 (0:00:00.092) 0:01:29.578 ******* 2020-11-29T15:52:50.456 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-29T15:52:50.458 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-29T15:52:50.493 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:50 +0000 (0:00:00.264) 0:01:29.842 ******* 2020-11-29T15:52:50.541 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-29T15:52:50.578 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:50 +0000 (0:00:00.085) 0:01:29.927 ******* 2020-11-29T15:52:51.385 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-29T15:52:51.387 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:52:51.421 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:51 +0000 (0:00:00.843) 0:01:30.770 ******* 2020-11-29T15:52:51.449 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:52:51.487 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:51 +0000 (0:00:00.065) 0:01:30.836 ******* 2020-11-29T15:52:51.599 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-29T15:52:51.601 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046488", "end": "2020-11-29 15:52:51.200647", "rc": 0, "start": "2020-11-29 15:52:51.154159", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-29T15:52:51.641 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:51 +0000 (0:00:00.154) 0:01:30.991 ******* 2020-11-29T15:52:51.669 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046079", "end": "2020-11-29 15:52:51.461028", "rc": 0, "start": "2020-11-29 15:52:51.414949", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-29T15:52:51.706 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:51 +0000 (0:00:00.064) 0:01:31.055 ******* 2020-11-29T15:52:51.712 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-29T15:52:51.714 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:51.749 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:51 +0000 (0:00:00.043) 0:01:31.098 ******* 2020-11-29T15:52:51.756 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:51.793 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:51 +0000 (0:00:00.043) 0:01:31.142 ******* 2020-11-29T15:52:51.799 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-29T15:52:51.801 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:51.837 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:51 +0000 (0:00:00.044) 0:01:31.187 ******* 2020-11-29T15:52:51.843 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:51.880 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:51 +0000 (0:00:00.042) 0:01:31.230 ******* 2020-11-29T15:52:52.651 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-29T15:52:52.652 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:52:52.688 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:52 +0000 (0:00:00.807) 0:01:32.037 ******* 2020-11-29T15:52:52.714 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-29T15:52:52.715 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:52.758 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:52 +0000 (0:00:00.070) 0:01:32.107 ******* 2020-11-29T15:52:52.766 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:52:52.812 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:52 +0000 (0:00:00.054) 0:01:32.161 ******* 2020-11-29T15:52:52.843 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-29T15:52:52.846 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:52.890 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:52 +0000 (0:00:00.077) 0:01:32.239 ******* 2020-11-29T15:52:53.111 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-29T15:52:53.113 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-29T15:52:53.170 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.279) 0:01:32.519 ******* 2020-11-29T15:52:53.214 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:53.280 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.109) 0:01:32.629 ******* 2020-11-29T15:52:53.290 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-29T15:52:53.292 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-29T15:52:53.344 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.064) 0:01:32.693 ******* 2020-11-29T15:52:53.374 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:53.407 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.062) 0:01:32.756 ******* 2020-11-29T15:52:53.496 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-29T15:52:53.497 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2020-11-29 15:50:04 UTC", "ActiveEnterTimestampMonotonic": "25216736", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice -.mount basic.target sysinit.target network.target systemd-journald.socket systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2020-11-29 15:49:58 UTC", "AssertTimestampMonotonic": "18692112", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2020-11-29 15:49:58 UTC", "ConditionTimestampMonotonic": "18692111", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-29T15:52:53.499 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1905", "ExecMainStartTimestamp": "Sun 2020-11-29 15:50:04 UTC", "ExecMainStartTimestampMonotonic": "25216723", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2020-11-29 15:49:58 UTC", "InactiveExitTimestampMonotonic": "18693037", "InvocationID": "135b329bfaad4498a3fdcb577a465e9f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-29T15:52:53.501 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1905", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2020-11-29 15:50:04 UTC", "S 2020-11-29T15:52:53.502 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "25216736", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2020-11-29 15:50:04 UTC", "WatchdogTimestampMonotonic": "25216735", "WatchdogUSec": "0"}} 2020-11-29T15:52:53.533 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.126) 0:01:32.882 ******* 2020-11-29T15:52:53.559 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:53.561 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-29T15:52:53.594 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.059) 0:01:32.942 ******* 2020-11-29T15:52:53.630 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:53.662 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.068) 0:01:33.011 ******* 2020-11-29T15:52:53.673 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-29T15:52:53.675 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2020-11-29 15:50:37 UTC", "ActiveEnterTimestampMonotonic": "24826469", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target -.mount basic.target systemd-tmpfiles-setup.service sysinit.target systemd-journald.socket system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2020-11-29 15:50:31 UTC", "AssertTimestampMonotonic": "18744077", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2020-11-29 15:50:31 UTC", "ConditionTimestampMonotonic": "18744076", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-29T15:52:53.677 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1893", "ExecMainStartTimestamp": "Sun 2020-11-29 15:50:37 UTC", "ExecMainStartTimestampMonotonic": "24826421", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2020-11-29 15:50:31 UTC", "InactiveExitTimestampMonotonic": "18744825", "InvocationID": "6845f17a90c34593937e833825b484cd", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-29T15:52:53.678 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1893", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2020-11-29 15:50:37 UTC", "S 2020-11-29T15:52:53.680 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24826469", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2020-11-29 15:50:37 UTC", "WatchdogTimestampMonotonic": "24826466", "WatchdogUSec": "0"}} 2020-11-29T15:52:53.713 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.051) 0:01:33.062 ******* 2020-11-29T15:52:53.721 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-29T15:52:53.722 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:53.760 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.046) 0:01:33.109 ******* 2020-11-29T15:52:53.766 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:53.767 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-29T15:52:53.807 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.047) 0:01:33.156 ******* 2020-11-29T15:52:53.835 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:53.837 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:53.876 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.068) 0:01:33.225 ******* 2020-11-29T15:52:53.902 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-29T15:52:53.905 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:53.939 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:53 +0000 (0:00:00.063) 0:01:33.288 ******* 2020-11-29T15:52:54.602 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-29T15:52:54.604 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1606665071, "cache_updated": false, "changed": false} 2020-11-29T15:52:54.657 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:54 +0000 (0:00:00.717) 0:01:34.006 ******* 2020-11-29T15:52:54.686 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:54.723 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:54 +0000 (0:00:00.066) 0:01:34.072 ******* 2020-11-29T15:52:54.846 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-29T15:52:54.848 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-29T15:52:54.908 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:54 +0000 (0:00:00.184) 0:01:34.257 ******* 2020-11-29T15:52:54.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-29T15:52:54.919 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"cache_update_time": 1606665067, "cache_updated": false, "changed": false} 2020-11-29T15:52:54.956 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:54 +0000 (0:00:00.047) 0:01:34.305 ******* 2020-11-29T15:52:54.983 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-29T15:52:54.985 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:55.019 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:55 +0000 (0:00:00.063) 0:01:34.368 ******* 2020-11-29T15:52:55.140 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-29T15:52:55.141 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "gid": 1235, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-29T15:52:55.191 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:55 +0000 (0:00:00.172) 0:01:34.540 ******* 2020-11-29T15:52:55.234 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1235, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-29T15:52:55.275 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:55 +0000 (0:00:00.083) 0:01:34.624 ******* 2020-11-29T15:52:55.588 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1235, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-29T15:52:55.625 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:55 +0000 (0:00:00.349) 0:01:34.974 ******* 2020-11-29T15:52:56.282 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-29T15:52:56.284 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-29T15:52:56.322 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:56 +0000 (0:00:00.697) 0:01:35.671 ******* 2020-11-29T15:52:56.370 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-29T15:52:56.413 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:56 +0000 (0:00:00.090) 0:01:35.762 ******* 2020-11-29T15:52:56.866 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-29T15:52:56.868 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false} 2020-11-29T15:52:56.902 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:56 +0000 (0:00:00.489) 0:01:36.251 ******* 2020-11-29T15:52:56.920 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false} 2020-11-29T15:52:56.960 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:56 +0000 (0:00:00.058) 0:01:36.309 ******* 2020-11-29T15:52:56.967 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-29T15:52:56.968 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-29T15:52:57.004 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.043) 0:01:36.353 ******* 2020-11-29T15:52:57.011 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-29T15:52:57.050 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.045) 0:01:36.399 ******* 2020-11-29T15:52:57.144 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-29T15:52:57.146 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1606665019.4260702, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1606665019.422068, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 113087, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1606665019.422068, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3531524185", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-29T15:52:57.185 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.135) 0:01:36.534 ******* 2020-11-29T15:52:57.203 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1606665052.4595437, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1606665052.4555416, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 113087, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1606665052.4555416, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3531524185", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-29T15:52:57.250 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.064) 0:01:36.599 ******* 2020-11-29T15:52:57.436 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-29T15:52:57.438 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi168.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-29T15:52:57.474 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.224) 0:01:36.823 ******* 2020-11-29T15:52:57.528 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.053) 0:01:36.877 ******* 2020-11-29T15:52:57.536 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-29T15:52:57.537 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-29T15:52:57.585 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.057) 0:01:36.934 ******* 2020-11-29T15:52:57.592 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-29T15:52:57.632 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.047) 0:01:36.981 ******* 2020-11-29T15:52:57.640 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-29T15:52:57.641 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:57.680 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.047) 0:01:37.029 ******* 2020-11-29T15:52:57.688 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:57.728 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.048) 0:01:37.078 ******* 2020-11-29T15:52:57.815 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-29T15:52:57.855 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.126) 0:01:37.205 ******* 2020-11-29T15:52:57.862 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-29T15:52:57.899 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:57 +0000 (0:00:00.043) 0:01:37.248 ******* 2020-11-29T15:52:57.980 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-29T15:52:57.981 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-29T15:52:58.031 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.132) 0:01:37.381 ******* 2020-11-29T15:52:58.041 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-29T15:52:58.083 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.051) 0:01:37.432 ******* 2020-11-29T15:52:58.090 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-29T15:52:58.091 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.138 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.054) 0:01:37.487 ******* 2020-11-29T15:52:58.144 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.193 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.055) 0:01:37.542 ******* 2020-11-29T15:52:58.199 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-29T15:52:58.201 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.201 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.251 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.057) 0:01:37.600 ******* 2020-11-29T15:52:58.257 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.259 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.299 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.048) 0:01:37.648 ******* 2020-11-29T15:52:58.305 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-29T15:52:58.307 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi168.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.319 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.330 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.331 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.368 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.069) 0:01:37.718 ******* 2020-11-29T15:52:58.375 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.377 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.415 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.046) 0:01:37.764 ******* 2020-11-29T15:52:58.421 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-29T15:52:58.423 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.460 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.044) 0:01:37.809 ******* 2020-11-29T15:52:58.466 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.504 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.044) 0:01:37.854 ******* 2020-11-29T15:52:58.510 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-29T15:52:58.512 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.550 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.045) 0:01:37.899 ******* 2020-11-29T15:52:58.556 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.594 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.044) 0:01:37.943 ******* 2020-11-29T15:52:58.601 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-29T15:52:58.603 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.640 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.046) 0:01:37.989 ******* 2020-11-29T15:52:58.648 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.688 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.048) 0:01:38.038 ******* 2020-11-29T15:52:58.695 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-29T15:52:58.697 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.734 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.045) 0:01:38.083 ******* 2020-11-29T15:52:58.741 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.784 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.049) 0:01:38.133 ******* 2020-11-29T15:52:58.789 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-29T15:52:58.791 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.851 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.067) 0:01:38.200 ******* 2020-11-29T15:52:58.857 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.898 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.047) 0:01:38.248 ******* 2020-11-29T15:52:58.904 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-29T15:52:58.906 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.949 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.050) 0:01:38.298 ******* 2020-11-29T15:52:58.957 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:58.997 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:58 +0000 (0:00:00.048) 0:01:38.346 ******* 2020-11-29T15:52:59.003 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-29T15:52:59.006 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:59.042 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:59 +0000 (0:00:00.045) 0:01:38.392 ******* 2020-11-29T15:52:59.049 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:59.102 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:59 +0000 (0:00:00.058) 0:01:38.451 ******* 2020-11-29T15:52:59.110 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-29T15:52:59.112 INFO:teuthology.task.ansible.out:skipping: [smithi168.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:59.152 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:59 +0000 (0:00:00.050) 0:01:38.501 ******* 2020-11-29T15:52:59.160 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-29T15:52:59.197 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:59 +0000 (0:00:00.045) 0:01:38.546 ******* 2020-11-29T15:52:59.311 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-29T15:52:59.312 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010428", "end": "2020-11-29 15:52:59.102808", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-29 15:52:59.092380", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:59.354 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011105", "end": "2020-11-29 15:52:58.945665", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-29 15:52:58.934560", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:59.438 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006040", "end": "2020-11-29 15:52:59.227733", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-29 15:52:59.221693", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:59.460 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005445", "end": "2020-11-29 15:52:59.064050", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-29 15:52:59.058605", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:59.557 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005457", "end": "2020-11-29 15:52:59.345788", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-29 15:52:59.340331", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:59.588 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005407", "end": "2020-11-29 15:52:59.186012", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-29 15:52:59.180605", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:59.695 INFO:teuthology.task.ansible.out:ok: [smithi168.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.022702", "end": "2020-11-29 15:52:59.476666", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-29 15:52:59.453964", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:59.739 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:59 +0000 (0:00:00.541) 0:01:39.088 ******* 2020-11-29T15:52:59.746 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.023434", "end": "2020-11-29 15:52:59.315575", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-29 15:52:59.292141", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-29T15:52:59.790 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:59 +0000 (0:00:00.050) 0:01:39.139 ******* 2020-11-29T15:52:59.797 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-29T15:52:59.842 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:59 +0000 (0:00:00.052) 0:01:39.192 ******* 2020-11-29T15:52:59.887 INFO:teuthology.task.ansible.out:Sunday 29 November 2020 15:52:59 +0000 (0:00:00.044) 0:01:39.236 ******* 2020-11-29T15:52:59.966 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-29T15:52:59.968 INFO:teuthology.task.ansible.out:changed: [smithi168.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-29T15:53:00.046 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-29T15:53:00.158 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-29T15:53:00.161 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-29T15:53:00.165 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* 2020-11-29T15:53:00.166 INFO:teuthology.task.ansible.out:smithi168.front.sepia.ceph.com : ok=114 changed=29 unreachable=0 failed=0 skipped=182 rescued=0 ignored=1 smithi184.front.sepia.ceph.com : ok=114 changed=29 unreachable=0 failed=0 skipped=182 rescued=0 ignored=1 Sunday 29 November 2020 15:53:00 +0000 (0:00:00.278) 0:01:39.514 ******* =============================================================================== testnode : Ensure packages are not present. ----------------------------- 9.11s testnode : Install packages --------------------------------------------- 8.73s testnode : Zap all non-root disks --------------------------------------- 8.44s common : Check for docker ----------------------------------------------- 4.47s common : Update apt cache ----------------------------------------------- 2.96s testnode : Update apt cache. -------------------------------------------- 2.78s testnode : Install apt keys --------------------------------------------- 2.12s Gathering Facts --------------------------------------------------------- 1.68s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.33s testnode : Create logical volume(s) ------------------------------------- 1.30s testnode : Install python-apt ------------------------------------------- 0.89s common : Install Kerberos Packages (Debian) ----------------------------- 0.89s testnode : Install non aarch64 packages --------------------------------- 0.88s testnode : Upgrade packages --------------------------------------------- 0.86s testnode : Make sure apt dependencies are installed --------------------- 0.85s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.84s testnode : Install ntp package on apt based systems. -------------------- 0.81s Gathering Facts --------------------------------------------------------- 0.80s users : Create all admin users with sudo access. ------------------------ 0.79s testnode : Set the system's hostname ------------ 2020-11-29T15:53:00.168 INFO:teuthology.task.ansible.out:------------------------ 0.76s 2020-11-29T15:53:00.486 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi168.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2020-11-29T15:53:00.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-29T15:53:00.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi168.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:53:00.561 INFO:teuthology.orchestra.run.smithi168:> true 2020-11-29T15:53:00.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-29T15:53:00.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-29T15:53:00.830 INFO:teuthology.orchestra.run.smithi184:> true 2020-11-29T15:53:01.030 INFO:teuthology.run_tasks:Running task clock... 2020-11-29T15:53:01.042 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-29T15:53:01.043 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-29T15:53:01.043 INFO:teuthology.orchestra.run.smithi168:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-29T15:53:01.101 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-29T15:53:01.101 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: Command line: ntpd -gq 2020-11-29T15:53:01.102 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: proto: precision = 0.123 usec (-23) 2020-11-29T15:53:01.103 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-29T15:53:01.103 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: restrict ::: KOD does nothing without LIMITED. 2020-11-29T15:53:01.103 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: Listen and drop on 0 v6wildcard [::]:123 2020-11-29T15:53:01.103 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-29T15:53:01.104 INFO:teuthology.orchestra.run.smithi168.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-29T15:53:01.104 INFO:teuthology.orchestra.run.smithi168.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-29T15:53:01.105 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: Listen normally on 2 lo 127.0.0.1:123 2020-11-29T15:53:01.105 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: Listen normally on 3 enp3s0f1 172.21.15.168:123 2020-11-29T15:53:01.105 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: Listen normally on 4 lo [::1]:123 2020-11-29T15:53:01.105 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:71fb%5]:123 2020-11-29T15:53:01.105 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:00 ntpd[7039]: Listening on routing socket on fd #22 for interface updates 2020-11-29T15:53:03.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:02 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:03.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:02 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:03.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:02 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:03.102 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:02 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:05.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:04 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:05.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:04 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:05.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:04 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:05.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:04 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:05.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:04 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:05.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:04 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:05.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:04 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:05.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:04 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:06.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:05 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:06.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:05 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:06.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:05 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:06.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:05 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:06.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:05 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:06.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:05 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:06.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:05 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:06.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:05 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:07.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:06 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:07.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:06 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:07.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:06 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:07.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:06 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:07.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:06 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:07.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:06 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:07.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:06 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:07.101 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:06 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:08.098 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:07 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:08.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:07 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:08.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:07 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:08.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:07 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:08.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:07 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:08.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:07 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:08.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:07 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:08.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:07 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:09.098 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:08 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:09.098 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:08 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:09.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:08 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:09.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:08 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:09.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:08 ntpd[7039]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:09.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:08 ntpd[7039]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:09.099 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:08 ntpd[7039]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:09.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:08 ntpd[7039]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:09.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:09 ntpd[7039]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-29T15:53:09.100 INFO:teuthology.orchestra.run.smithi168.stderr:29 Nov 15:53:09 ntpd[7039]: can't open /var/log/ntpstats/loopstats.20201129: Permission denied 2020-11-29T15:53:09.100 INFO:teuthology.orchestra.run.smithi168.stdout:29 Nov 15:53:09 ntpd[7039]: ntpd: time set +0.177476 s 2020-11-29T15:53:09.100 INFO:teuthology.orchestra.run.smithi168.stdout:ntpd: time set +0.177476s 2020-11-29T15:53:09.148 INFO:teuthology.orchestra.run.smithi168.stdout: remote refid st t when poll reach delay offset jitter 2020-11-29T15:53:09.149 INFO:teuthology.orchestra.run.smithi168.stdout:============================================================================== 2020-11-29T15:53:09.149 INFO:teuthology.orchestra.run.smithi168.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-29T15:53:09.150 INFO:teuthology.orchestra.run.smithi168.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-29T15:53:09.150 INFO:teuthology.orchestra.run.smithi168.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-29T15:53:09.151 INFO:teuthology.orchestra.run.smithi168.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-29T15:53:09.151 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-29T15:53:09.152 INFO:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-29T15:53:09.213 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-29T15:53:09.213 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: Command line: ntpd -gq 2020-11-29T15:53:09.216 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: proto: precision = 0.120 usec (-23) 2020-11-29T15:53:09.216 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-29T15:53:09.217 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: restrict ::: KOD does nothing without LIMITED. 2020-11-29T15:53:09.217 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: Listen and drop on 0 v6wildcard [::]:123 2020-11-29T15:53:09.217 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-29T15:53:09.218 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: Listen normally on 2 lo 127.0.0.1:123 2020-11-29T15:53:09.218 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: Listen normally on 3 enp3s0f1 172.21.15.184:123 2020-11-29T15:53:09.218 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: Listen normally on 4 lo [::1]:123 2020-11-29T15:53:09.218 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7b8f%5]:123 2020-11-29T15:53:09.219 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:08 ntpd[7078]: Listening on routing socket on fd #22 for interface updates 2020-11-29T15:53:09.219 INFO:teuthology.orchestra.run.smithi184.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-29T15:53:09.219 INFO:teuthology.orchestra.run.smithi184.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-29T15:53:10.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:10.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:10.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:10.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:10.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:10.215 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:10.215 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:10.215 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:10.215 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:10.215 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:10.216 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:10.216 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:09 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:12.212 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:12.212 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:12.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:12.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:12.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:12.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:12.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:12.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:12.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:12.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:12.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:12.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:11 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:14.211 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:14.212 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:14.212 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:14.212 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:14.212 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:14.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:14.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:14.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:14.213 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:14.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:14.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:14.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:14.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:14.214 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:14.215 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:14.215 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:13 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:16.210 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:15 ntpd[7078]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-29T15:53:16.210 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:15 ntpd[7078]: can't open /var/log/ntpstats/rawstats.20201129: Permission denied 2020-11-29T15:53:16.210 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:15 ntpd[7078]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-29T15:53:16.210 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:15 ntpd[7078]: can't open /var/log/ntpstats/peerstats.20201129: Permission denied 2020-11-29T15:53:16.211 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:16 ntpd[7078]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-29T15:53:16.211 INFO:teuthology.orchestra.run.smithi184.stderr:29 Nov 15:53:16 ntpd[7078]: can't open /var/log/ntpstats/loopstats.20201129: Permission denied 2020-11-29T15:53:16.211 INFO:teuthology.orchestra.run.smithi184.stdout:29 Nov 15:53:16 ntpd[7078]: ntpd: time set +0.365284 s 2020-11-29T15:53:16.211 INFO:teuthology.orchestra.run.smithi184.stdout:ntpd: time set +0.365284s 2020-11-29T15:53:16.258 INFO:teuthology.orchestra.run.smithi184.stdout: remote refid st t when poll reach delay offset jitter 2020-11-29T15:53:16.259 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================== 2020-11-29T15:53:16.259 INFO:teuthology.orchestra.run.smithi184.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-29T15:53:16.259 INFO:teuthology.orchestra.run.smithi184.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-29T15:53:16.259 INFO:teuthology.orchestra.run.smithi184.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-29T15:53:16.260 INFO:teuthology.orchestra.run.smithi184.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-29T15:53:16.260 INFO:teuthology.run_tasks:Running task install... 2020-11-29T15:53:16.273 DEBUG:teuthology.task.install:project ceph 2020-11-29T15:53:16.274 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6'}} 2020-11-29T15:53:16.274 DEBUG:teuthology.task.install:config {'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6'} 2020-11-29T15:53:16.274 INFO:teuthology.task.install:Using flavor: basic 2020-11-29T15:53:16.282 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-29T15:53:16.282 INFO:teuthology.task.install:extra packages: [] 2020-11-29T15:53:16.283 INFO:teuthology.orchestra.run.smithi168:> sudo lsb_release -is 2020-11-29T15:53:16.337 INFO:teuthology.orchestra.run.smithi168.stdout:Ubuntu 2020-11-29T15:53:16.342 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-29T15:53:16.342 INFO:teuthology.orchestra.run.smithi184:> sudo lsb_release -is 2020-11-29T15:53:16.343 INFO:teuthology.orchestra.run.smithi168:> sudo apt-key list | grep Ceph 2020-11-29T15:53:16.396 INFO:teuthology.orchestra.run.smithi184.stdout:Ubuntu 2020-11-29T15:53:16.401 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-29T15:53:16.402 INFO:teuthology.orchestra.run.smithi184:> sudo apt-key list | grep Ceph 2020-11-29T15:53:16.434 INFO:teuthology.orchestra.run.smithi168.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-29T15:53:16.494 INFO:teuthology.orchestra.run.smithi168.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-29T15:53:16.494 INFO:teuthology.orchestra.run.smithi168.stdout:uid [ unknown] Ceph.com (release key) 2020-11-29T15:53:16.495 INFO:teuthology.orchestra.run.smithi168:> sudo lsb_release -is 2020-11-29T15:53:16.496 INFO:teuthology.orchestra.run.smithi184.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-29T15:53:16.545 INFO:teuthology.orchestra.run.smithi168.stdout:Ubuntu 2020-11-29T15:53:16.550 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-29T15:53:16.550 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-29T15:53:16.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=10684a45d9b8cdfc272d8721720e890aa44052c6 2020-11-29T15:53:16.555 INFO:teuthology.orchestra.run.smithi184.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-29T15:53:16.555 INFO:teuthology.orchestra.run.smithi184.stdout:uid [ unknown] Ceph.com (release key) 2020-11-29T15:53:16.556 INFO:teuthology.orchestra.run.smithi184:> sudo lsb_release -is 2020-11-29T15:53:16.606 INFO:teuthology.orchestra.run.smithi184.stdout:Ubuntu 2020-11-29T15:53:16.611 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-29T15:53:16.612 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-11-29T15:53:16.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=10684a45d9b8cdfc272d8721720e890aa44052c6 2020-11-29T15:53:16.715 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default/ 2020-11-29T15:53:16.716 INFO:teuthology.task.install.deb:Package version is 16.0.0-7606-g10684a45-1bionic 2020-11-29T15:53:16.771 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default/ 2020-11-29T15:53:16.771 INFO:teuthology.task.install.deb:Package version is 16.0.0-7606-g10684a45-1bionic 2020-11-29T15:53:16.849 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:53:16.850 INFO:teuthology.orchestra.run.smithi168:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-29T15:53:16.855 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-29T15:53:16.864 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:53:16.865 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:53:16.865 INFO:teuthology.orchestra.run.smithi168.stderr:169 bytes copied, 7.3616e-05 s, 2.3 MB/s 2020-11-29T15:53:16.866 INFO:teuthology.orchestra.run.smithi168:> sudo apt-get update 2020-11-29T15:53:16.895 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:53:16.896 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-29T15:53:16.900 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-29T15:53:16.909 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:53:16.909 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:53:16.909 INFO:teuthology.orchestra.run.smithi184.stderr:169 bytes copied, 7.04e-05 s, 2.4 MB/s 2020-11-29T15:53:16.911 INFO:teuthology.orchestra.run.smithi184:> sudo apt-get update 2020-11-29T15:53:17.070 INFO:teuthology.orchestra.run.smithi168.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-29T15:53:17.099 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-29T15:53:17.121 INFO:teuthology.orchestra.run.smithi168.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic InRelease 2020-11-29T15:53:17.144 INFO:teuthology.orchestra.run.smithi168.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-29T15:53:17.156 INFO:teuthology.orchestra.run.smithi168.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-29T15:53:17.169 INFO:teuthology.orchestra.run.smithi168.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-29T15:53:17.181 INFO:teuthology.orchestra.run.smithi184.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic InRelease 2020-11-29T15:53:17.194 INFO:teuthology.orchestra.run.smithi168.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,948 B] 2020-11-29T15:53:17.201 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-29T15:53:17.204 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-29T15:53:17.230 INFO:teuthology.orchestra.run.smithi184.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-29T15:53:17.235 INFO:teuthology.orchestra.run.smithi168.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-29T15:53:17.243 INFO:teuthology.orchestra.run.smithi168.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-29T15:53:17.254 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-29T15:53:17.280 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-29T15:53:17.296 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,948 B] 2020-11-29T15:53:17.314 INFO:teuthology.orchestra.run.smithi168.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-29T15:53:17.358 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-29T15:53:17.688 INFO:teuthology.orchestra.run.smithi168.stdout:Fetched 26.7 kB in 1s (42.5 kB/s) 2020-11-29T15:53:17.700 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 26.7 kB in 1s (45.1 kB/s) 2020-11-29T15:53:18.921 INFO:teuthology.orchestra.run.smithi168.stdout:Reading package lists... 2020-11-29T15:53:18.932 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-11-29T15:53:18.966 INFO:teuthology.orchestra.run.smithi168:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7606-g10684a45-1bionic cephadm=16.0.0-7606-g10684a45-1bionic ceph-mds=16.0.0-7606-g10684a45-1bionic ceph-mgr=16.0.0-7606-g10684a45-1bionic ceph-common=16.0.0-7606-g10684a45-1bionic ceph-fuse=16.0.0-7606-g10684a45-1bionic ceph-test=16.0.0-7606-g10684a45-1bionic radosgw=16.0.0-7606-g10684a45-1bionic python3-rados=16.0.0-7606-g10684a45-1bionic python3-rgw=16.0.0-7606-g10684a45-1bionic python3-cephfs=16.0.0-7606-g10684a45-1bionic python3-rbd=16.0.0-7606-g10684a45-1bionic libcephfs2=16.0.0-7606-g10684a45-1bionic libcephfs-dev=16.0.0-7606-g10684a45-1bionic librados2=16.0.0-7606-g10684a45-1bionic librbd1=16.0.0-7606-g10684a45-1bionic rbd-fuse=16.0.0-7606-g10684a45-1bionic 2020-11-29T15:53:18.979 INFO:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7606-g10684a45-1bionic cephadm=16.0.0-7606-g10684a45-1bionic ceph-mds=16.0.0-7606-g10684a45-1bionic ceph-mgr=16.0.0-7606-g10684a45-1bionic ceph-common=16.0.0-7606-g10684a45-1bionic ceph-fuse=16.0.0-7606-g10684a45-1bionic ceph-test=16.0.0-7606-g10684a45-1bionic radosgw=16.0.0-7606-g10684a45-1bionic python3-rados=16.0.0-7606-g10684a45-1bionic python3-rgw=16.0.0-7606-g10684a45-1bionic python3-cephfs=16.0.0-7606-g10684a45-1bionic python3-rbd=16.0.0-7606-g10684a45-1bionic libcephfs2=16.0.0-7606-g10684a45-1bionic libcephfs-dev=16.0.0-7606-g10684a45-1bionic librados2=16.0.0-7606-g10684a45-1bionic librbd1=16.0.0-7606-g10684a45-1bionic rbd-fuse=16.0.0-7606-g10684a45-1bionic 2020-11-29T15:53:19.042 INFO:teuthology.orchestra.run.smithi168.stdout:Reading package lists... 2020-11-29T15:53:19.060 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-11-29T15:53:19.271 INFO:teuthology.orchestra.run.smithi168.stdout:Building dependency tree... 2020-11-29T15:53:19.271 INFO:teuthology.orchestra.run.smithi168.stdout:Reading state information... 2020-11-29T15:53:19.309 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2020-11-29T15:53:19.310 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2020-11-29T15:53:19.433 INFO:teuthology.orchestra.run.smithi168.stdout:The following additional packages will be installed: 2020-11-29T15:53:19.433 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-29T15:53:19.434 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-29T15:53:19.435 INFO:teuthology.orchestra.run.smithi168.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-29T15:53:19.435 INFO:teuthology.orchestra.run.smithi168.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-29T15:53:19.435 INFO:teuthology.orchestra.run.smithi168.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-29T15:53:19.435 INFO:teuthology.orchestra.run.smithi168.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-29T15:53:19.435 INFO:teuthology.orchestra.run.smithi168.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-29T15:53:19.436 INFO:teuthology.orchestra.run.smithi168.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-29T15:53:19.436 INFO:teuthology.orchestra.run.smithi168.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-29T15:53:19.436 INFO:teuthology.orchestra.run.smithi168.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-29T15:53:19.436 INFO:teuthology.orchestra.run.smithi168.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-29T15:53:19.436 INFO:teuthology.orchestra.run.smithi168.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-29T15:53:19.437 INFO:teuthology.orchestra.run.smithi168.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-29T15:53:19.437 INFO:teuthology.orchestra.run.smithi168.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-29T15:53:19.437 INFO:teuthology.orchestra.run.smithi168.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-29T15:53:19.437 INFO:teuthology.orchestra.run.smithi168.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-29T15:53:19.438 INFO:teuthology.orchestra.run.smithi168.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-29T15:53:19.438 INFO:teuthology.orchestra.run.smithi168.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-29T15:53:19.438 INFO:teuthology.orchestra.run.smithi168.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-29T15:53:19.438 INFO:teuthology.orchestra.run.smithi168.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-29T15:53:19.438 INFO:teuthology.orchestra.run.smithi168.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-29T15:53:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-29T15:53:19.439 INFO:teuthology.orchestra.run.smithi168.stdout:Suggested packages: 2020-11-29T15:53:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-29T15:53:19.439 INFO:teuthology.orchestra.run.smithi168.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-29T15:53:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-29T15:53:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-29T15:53:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-29T15:53:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-29T15:53:19.440 INFO:teuthology.orchestra.run.smithi168.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-29T15:53:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-29T15:53:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-29T15:53:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-29T15:53:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-29T15:53:19.441 INFO:teuthology.orchestra.run.smithi168.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-29T15:53:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-29T15:53:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-29T15:53:19.442 INFO:teuthology.orchestra.run.smithi168.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-29T15:53:19.472 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2020-11-29T15:53:19.472 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-29T15:53:19.473 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-29T15:53:19.473 INFO:teuthology.orchestra.run.smithi184.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-29T15:53:19.474 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-29T15:53:19.474 INFO:teuthology.orchestra.run.smithi184.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-29T15:53:19.474 INFO:teuthology.orchestra.run.smithi184.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-29T15:53:19.474 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-29T15:53:19.474 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-11-29T15:53:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-11-29T15:53:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-11-29T15:53:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-29T15:53:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-29T15:53:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-29T15:53:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-29T15:53:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-29T15:53:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-11-29T15:53:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-29T15:53:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-29T15:53:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-29T15:53:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-29T15:53:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-29T15:53:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-29T15:53:19.478 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2020-11-29T15:53:19.478 INFO:teuthology.orchestra.run.smithi184.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-29T15:53:19.478 INFO:teuthology.orchestra.run.smithi184.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-29T15:53:19.478 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-29T15:53:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-29T15:53:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-29T15:53:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-29T15:53:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-29T15:53:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-29T15:53:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-29T15:53:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-29T15:53:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-29T15:53:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-29T15:53:19.481 INFO:teuthology.orchestra.run.smithi184.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-29T15:53:19.481 INFO:teuthology.orchestra.run.smithi184.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-29T15:53:19.481 INFO:teuthology.orchestra.run.smithi184.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-29T15:53:19.540 INFO:teuthology.orchestra.run.smithi168.stdout:The following NEW packages will be installed: 2020-11-29T15:53:19.540 INFO:teuthology.orchestra.run.smithi168.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-29T15:53:19.541 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-29T15:53:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-29T15:53:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-29T15:53:19.542 INFO:teuthology.orchestra.run.smithi168.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-29T15:53:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-29T15:53:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-29T15:53:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-29T15:53:19.543 INFO:teuthology.orchestra.run.smithi168.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-29T15:53:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-29T15:53:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-29T15:53:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-29T15:53:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-29T15:53:19.544 INFO:teuthology.orchestra.run.smithi168.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-29T15:53:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-29T15:53:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-29T15:53:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-29T15:53:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-29T15:53:19.545 INFO:teuthology.orchestra.run.smithi168.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-29T15:53:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-29T15:53:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-29T15:53:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-29T15:53:19.546 INFO:teuthology.orchestra.run.smithi168.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-29T15:53:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-29T15:53:19.547 INFO:teuthology.orchestra.run.smithi168.stdout:The following packages will be upgraded: 2020-11-29T15:53:19.547 INFO:teuthology.orchestra.run.smithi168.stdout: librados2 librbd1 2020-11-29T15:53:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2020-11-29T15:53:19.572 INFO:teuthology.orchestra.run.smithi184.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-29T15:53:19.573 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-29T15:53:19.574 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-11-29T15:53:19.574 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-11-29T15:53:19.574 INFO:teuthology.orchestra.run.smithi184.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-11-29T15:53:19.574 INFO:teuthology.orchestra.run.smithi184.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-11-29T15:53:19.575 INFO:teuthology.orchestra.run.smithi184.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-11-29T15:53:19.575 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-11-29T15:53:19.575 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-11-29T15:53:19.575 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-29T15:53:19.576 INFO:teuthology.orchestra.run.smithi184.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-29T15:53:19.576 INFO:teuthology.orchestra.run.smithi184.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-29T15:53:19.576 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-29T15:53:19.576 INFO:teuthology.orchestra.run.smithi184.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-29T15:53:19.576 INFO:teuthology.orchestra.run.smithi184.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-29T15:53:19.577 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-29T15:53:19.577 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-29T15:53:19.577 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-11-29T15:53:19.577 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-11-29T15:53:19.577 INFO:teuthology.orchestra.run.smithi184.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-29T15:53:19.578 INFO:teuthology.orchestra.run.smithi184.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-29T15:53:19.578 INFO:teuthology.orchestra.run.smithi184.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-29T15:53:19.578 INFO:teuthology.orchestra.run.smithi184.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-29T15:53:19.578 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-29T15:53:19.579 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages will be upgraded: 2020-11-29T15:53:19.579 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 librbd1 2020-11-29T15:53:19.704 INFO:teuthology.orchestra.run.smithi168.stdout:2 upgraded, 110 newly installed, 0 to remove and 0 not upgraded. 2020-11-29T15:53:19.704 INFO:teuthology.orchestra.run.smithi168.stdout:Need to get 169 MB of archives. 2020-11-29T15:53:19.704 INFO:teuthology.orchestra.run.smithi168.stdout:After this operation, 792 MB of additional disk space will be used. 2020-11-29T15:53:19.705 INFO:teuthology.orchestra.run.smithi168.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7606-g10684a45-1bionic [2,212 kB] 2020-11-29T15:53:19.729 INFO:teuthology.orchestra.run.smithi168.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-29T15:53:19.739 INFO:teuthology.orchestra.run.smithi184.stdout:2 upgraded, 110 newly installed, 0 to remove and 0 not upgraded. 2020-11-29T15:53:19.739 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 169 MB of archives. 2020-11-29T15:53:19.739 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 792 MB of additional disk space will be used. 2020-11-29T15:53:19.739 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7606-g10684a45-1bionic [2,212 kB] 2020-11-29T15:53:19.761 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-29T15:53:19.809 INFO:teuthology.orchestra.run.smithi168.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-29T15:53:19.840 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-29T15:53:19.960 INFO:teuthology.orchestra.run.smithi168.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7606-g10684a45-1bionic [3,155 kB] 2020-11-29T15:53:19.995 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7606-g10684a45-1bionic [3,155 kB] 2020-11-29T15:53:20.068 INFO:teuthology.orchestra.run.smithi168.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-29T15:53:20.074 INFO:teuthology.orchestra.run.smithi168.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-29T15:53:20.085 INFO:teuthology.orchestra.run.smithi168.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-29T15:53:20.094 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-29T15:53:20.101 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-29T15:53:20.111 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-29T15:53:20.144 INFO:teuthology.orchestra.run.smithi168.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7606-g10684a45-1bionic [477 kB] 2020-11-29T15:53:20.176 INFO:teuthology.orchestra.run.smithi168.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7606-g10684a45-1bionic [332 kB] 2020-11-29T15:53:20.187 INFO:teuthology.orchestra.run.smithi168.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-29T15:53:20.190 INFO:teuthology.orchestra.run.smithi168.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-29T15:53:20.193 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7606-g10684a45-1bionic [477 kB] 2020-11-29T15:53:20.203 INFO:teuthology.orchestra.run.smithi168.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7606-g10684a45-1bionic [20.6 kB] 2020-11-29T15:53:20.204 INFO:teuthology.orchestra.run.smithi168.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7606-g10684a45-1bionic [168 kB] 2020-11-29T15:53:20.208 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-29T15:53:20.209 INFO:teuthology.orchestra.run.smithi168.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7606-g10684a45-1bionic [27.7 kB] 2020-11-29T15:53:20.210 INFO:teuthology.orchestra.run.smithi168.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7606-g10684a45-1bionic [330 kB] 2020-11-29T15:53:20.211 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-29T15:53:20.223 INFO:teuthology.orchestra.run.smithi168.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-29T15:53:20.227 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7606-g10684a45-1bionic [332 kB] 2020-11-29T15:53:20.231 INFO:teuthology.orchestra.run.smithi168.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-29T15:53:20.238 INFO:teuthology.orchestra.run.smithi168.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7606-g10684a45-1bionic [3,187 kB] 2020-11-29T15:53:20.239 INFO:teuthology.orchestra.run.smithi168.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-29T15:53:20.251 INFO:teuthology.orchestra.run.smithi168.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-29T15:53:20.257 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7606-g10684a45-1bionic [20.6 kB] 2020-11-29T15:53:20.258 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7606-g10684a45-1bionic [168 kB] 2020-11-29T15:53:20.260 INFO:teuthology.orchestra.run.smithi168.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-29T15:53:20.263 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-29T15:53:20.263 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7606-g10684a45-1bionic [27.7 kB] 2020-11-29T15:53:20.264 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7606-g10684a45-1bionic [330 kB] 2020-11-29T15:53:20.268 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-29T15:53:20.276 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-29T15:53:20.286 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-29T15:53:20.287 INFO:teuthology.orchestra.run.smithi168.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-29T15:53:20.291 INFO:teuthology.orchestra.run.smithi168.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-29T15:53:20.293 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-29T15:53:20.295 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7606-g10684a45-1bionic [3,187 kB] 2020-11-29T15:53:20.309 INFO:teuthology.orchestra.run.smithi168.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-29T15:53:20.316 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-29T15:53:20.320 INFO:teuthology.orchestra.run.smithi168.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-29T15:53:20.321 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-29T15:53:20.321 INFO:teuthology.orchestra.run.smithi168.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-29T15:53:20.331 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-29T15:53:20.346 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-29T15:53:20.389 INFO:teuthology.orchestra.run.smithi168.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-29T15:53:20.390 INFO:teuthology.orchestra.run.smithi168.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-29T15:53:20.392 INFO:teuthology.orchestra.run.smithi168.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-29T15:53:20.393 INFO:teuthology.orchestra.run.smithi168.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-29T15:53:20.394 INFO:teuthology.orchestra.run.smithi168.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-29T15:53:20.396 INFO:teuthology.orchestra.run.smithi168.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-29T15:53:20.398 INFO:teuthology.orchestra.run.smithi168.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-29T15:53:20.409 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-29T15:53:20.410 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-29T15:53:20.411 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-29T15:53:20.413 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-29T15:53:20.414 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-29T15:53:20.415 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-29T15:53:20.417 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-29T15:53:20.420 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-29T15:53:20.443 INFO:teuthology.orchestra.run.smithi168.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7606-g10684a45-1bionic [99.3 kB] 2020-11-29T15:53:20.446 INFO:teuthology.orchestra.run.smithi168.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7606-g10684a45-1bionic [412 kB] 2020-11-29T15:53:20.461 INFO:teuthology.orchestra.run.smithi168.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7606-g10684a45-1bionic [20.5 MB] 2020-11-29T15:53:20.468 INFO:teuthology.orchestra.run.smithi168.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-29T15:53:20.469 INFO:teuthology.orchestra.run.smithi168.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-29T15:53:20.487 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-29T15:53:20.488 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-29T15:53:20.490 INFO:teuthology.orchestra.run.smithi168.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-29T15:53:20.514 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7606-g10684a45-1bionic [99.3 kB] 2020-11-29T15:53:20.519 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7606-g10684a45-1bionic [412 kB] 2020-11-29T15:53:20.531 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7606-g10684a45-1bionic [20.5 MB] 2020-11-29T15:53:20.548 INFO:teuthology.orchestra.run.smithi168.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-29T15:53:20.549 INFO:teuthology.orchestra.run.smithi168.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-29T15:53:20.550 INFO:teuthology.orchestra.run.smithi168.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-29T15:53:20.554 INFO:teuthology.orchestra.run.smithi168.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-29T15:53:20.562 INFO:teuthology.orchestra.run.smithi168.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-29T15:53:20.566 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-29T15:53:20.566 INFO:teuthology.orchestra.run.smithi168.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-29T15:53:20.567 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-29T15:53:20.567 INFO:teuthology.orchestra.run.smithi168.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-29T15:53:20.568 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-29T15:53:20.568 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-29T15:53:20.571 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-29T15:53:20.579 INFO:teuthology.orchestra.run.smithi184.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-29T15:53:20.582 INFO:teuthology.orchestra.run.smithi184.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-29T15:53:20.583 INFO:teuthology.orchestra.run.smithi184.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-29T15:53:20.628 INFO:teuthology.orchestra.run.smithi168.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-29T15:53:20.644 INFO:teuthology.orchestra.run.smithi184.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-29T15:53:20.717 INFO:teuthology.orchestra.run.smithi168.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-29T15:53:20.718 INFO:teuthology.orchestra.run.smithi168.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-29T15:53:20.725 INFO:teuthology.orchestra.run.smithi184.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-29T15:53:20.725 INFO:teuthology.orchestra.run.smithi184.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-29T15:53:20.997 INFO:teuthology.orchestra.run.smithi168.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-29T15:53:21.007 INFO:teuthology.orchestra.run.smithi184.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-29T15:53:21.038 INFO:teuthology.orchestra.run.smithi168.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-29T15:53:21.039 INFO:teuthology.orchestra.run.smithi168.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-29T15:53:21.044 INFO:teuthology.orchestra.run.smithi184.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-29T15:53:21.047 INFO:teuthology.orchestra.run.smithi184.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-29T15:53:21.075 INFO:teuthology.orchestra.run.smithi168.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-29T15:53:21.075 INFO:teuthology.orchestra.run.smithi168.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-29T15:53:21.075 INFO:teuthology.orchestra.run.smithi168.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-29T15:53:21.076 INFO:teuthology.orchestra.run.smithi168.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-29T15:53:21.076 INFO:teuthology.orchestra.run.smithi168.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-29T15:53:21.077 INFO:teuthology.orchestra.run.smithi184.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-29T15:53:21.078 INFO:teuthology.orchestra.run.smithi168.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-29T15:53:21.078 INFO:teuthology.orchestra.run.smithi184.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-29T15:53:21.078 INFO:teuthology.orchestra.run.smithi184.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-29T15:53:21.079 INFO:teuthology.orchestra.run.smithi168.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-29T15:53:21.079 INFO:teuthology.orchestra.run.smithi184.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-29T15:53:21.081 INFO:teuthology.orchestra.run.smithi184.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-29T15:53:21.082 INFO:teuthology.orchestra.run.smithi184.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-29T15:53:21.087 INFO:teuthology.orchestra.run.smithi184.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-29T15:53:21.156 INFO:teuthology.orchestra.run.smithi168.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-29T15:53:21.158 INFO:teuthology.orchestra.run.smithi168.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-29T15:53:21.161 INFO:teuthology.orchestra.run.smithi184.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-29T15:53:21.164 INFO:teuthology.orchestra.run.smithi184.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-29T15:53:21.165 INFO:teuthology.orchestra.run.smithi168.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-29T15:53:21.167 INFO:teuthology.orchestra.run.smithi168.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-29T15:53:21.171 INFO:teuthology.orchestra.run.smithi184.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-29T15:53:21.172 INFO:teuthology.orchestra.run.smithi168.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-29T15:53:21.172 INFO:teuthology.orchestra.run.smithi184.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-29T15:53:21.175 INFO:teuthology.orchestra.run.smithi184.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-29T15:53:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-29T15:53:21.176 INFO:teuthology.orchestra.run.smithi184.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-29T15:53:21.177 INFO:teuthology.orchestra.run.smithi184.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-29T15:53:21.178 INFO:teuthology.orchestra.run.smithi184.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-29T15:53:21.236 INFO:teuthology.orchestra.run.smithi168.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-29T15:53:21.237 INFO:teuthology.orchestra.run.smithi168.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-29T15:53:21.241 INFO:teuthology.orchestra.run.smithi168.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-29T15:53:21.244 INFO:teuthology.orchestra.run.smithi168.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-29T15:53:21.245 INFO:teuthology.orchestra.run.smithi184.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-29T15:53:21.246 INFO:teuthology.orchestra.run.smithi168.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-29T15:53:21.255 INFO:teuthology.orchestra.run.smithi184.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-29T15:53:21.315 INFO:teuthology.orchestra.run.smithi168.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-29T15:53:21.319 INFO:teuthology.orchestra.run.smithi168.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-29T15:53:21.324 INFO:teuthology.orchestra.run.smithi184.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-29T15:53:21.325 INFO:teuthology.orchestra.run.smithi168.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2020-11-29T15:53:21.330 INFO:teuthology.orchestra.run.smithi168.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-29T15:53:21.330 INFO:teuthology.orchestra.run.smithi168.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-29T15:53:21.335 INFO:teuthology.orchestra.run.smithi184.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2020-11-29T15:53:21.337 INFO:teuthology.orchestra.run.smithi184.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-29T15:53:21.338 INFO:teuthology.orchestra.run.smithi184.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-29T15:53:21.338 INFO:teuthology.orchestra.run.smithi184.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-29T15:53:21.338 INFO:teuthology.orchestra.run.smithi184.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-29T15:53:21.339 INFO:teuthology.orchestra.run.smithi184.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-29T15:53:21.340 INFO:teuthology.orchestra.run.smithi184.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-29T15:53:21.395 INFO:teuthology.orchestra.run.smithi168.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-29T15:53:21.396 INFO:teuthology.orchestra.run.smithi168.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-29T15:53:21.398 INFO:teuthology.orchestra.run.smithi168.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-29T15:53:21.402 INFO:teuthology.orchestra.run.smithi184.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-29T15:53:21.403 INFO:teuthology.orchestra.run.smithi168.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-29T15:53:21.403 INFO:teuthology.orchestra.run.smithi184.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-29T15:53:21.404 INFO:teuthology.orchestra.run.smithi168.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-29T15:53:21.475 INFO:teuthology.orchestra.run.smithi168.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-29T15:53:21.476 INFO:teuthology.orchestra.run.smithi168.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-29T15:53:21.478 INFO:teuthology.orchestra.run.smithi168.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-29T15:53:21.481 INFO:teuthology.orchestra.run.smithi184.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-29T15:53:21.484 INFO:teuthology.orchestra.run.smithi184.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-29T15:53:21.498 INFO:teuthology.orchestra.run.smithi168.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-29T15:53:21.498 INFO:teuthology.orchestra.run.smithi168.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-29T15:53:21.506 INFO:teuthology.orchestra.run.smithi184.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-29T15:53:21.506 INFO:teuthology.orchestra.run.smithi184.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-29T15:53:21.507 INFO:teuthology.orchestra.run.smithi184.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-29T15:53:21.554 INFO:teuthology.orchestra.run.smithi168.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-29T15:53:21.646 INFO:teuthology.orchestra.run.smithi184.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-29T15:53:21.647 INFO:teuthology.orchestra.run.smithi184.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-29T15:53:21.648 INFO:teuthology.orchestra.run.smithi184.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-29T15:53:21.654 INFO:teuthology.orchestra.run.smithi184.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-29T15:53:21.654 INFO:teuthology.orchestra.run.smithi184.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-29T15:53:21.655 INFO:teuthology.orchestra.run.smithi184.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-29T15:53:21.655 INFO:teuthology.orchestra.run.smithi184.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-29T15:53:21.656 INFO:teuthology.orchestra.run.smithi184.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-29T15:53:21.657 INFO:teuthology.orchestra.run.smithi184.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-29T15:53:21.658 INFO:teuthology.orchestra.run.smithi168.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-29T15:53:21.658 INFO:teuthology.orchestra.run.smithi168.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-29T15:53:21.659 INFO:teuthology.orchestra.run.smithi168.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-29T15:53:21.663 INFO:teuthology.orchestra.run.smithi168.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-29T15:53:21.663 INFO:teuthology.orchestra.run.smithi168.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-29T15:53:21.664 INFO:teuthology.orchestra.run.smithi168.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-29T15:53:21.664 INFO:teuthology.orchestra.run.smithi168.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-29T15:53:21.666 INFO:teuthology.orchestra.run.smithi168.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-29T15:53:21.666 INFO:teuthology.orchestra.run.smithi168.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-29T15:53:21.741 INFO:teuthology.orchestra.run.smithi168.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7606-g10684a45-1bionic [5,600 kB] 2020-11-29T15:53:21.954 INFO:teuthology.orchestra.run.smithi184.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7606-g10684a45-1bionic [5,600 kB] 2020-11-29T15:53:22.108 INFO:teuthology.orchestra.run.smithi168.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7606-g10684a45-1bionic [165 kB] 2020-11-29T15:53:22.113 INFO:teuthology.orchestra.run.smithi168.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7606-g10684a45-1bionic [1,358 kB] 2020-11-29T15:53:22.189 INFO:teuthology.orchestra.run.smithi168.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7606-g10684a45-1bionic [6,658 kB] 2020-11-29T15:53:22.344 INFO:teuthology.orchestra.run.smithi184.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7606-g10684a45-1bionic [165 kB] 2020-11-29T15:53:22.349 INFO:teuthology.orchestra.run.smithi184.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7606-g10684a45-1bionic [1,358 kB] 2020-11-29T15:53:22.462 INFO:teuthology.orchestra.run.smithi184.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7606-g10684a45-1bionic [6,658 kB] 2020-11-29T15:53:22.611 INFO:teuthology.orchestra.run.smithi168.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7606-g10684a45-1bionic [24.6 MB] 2020-11-29T15:53:22.901 INFO:teuthology.orchestra.run.smithi184.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7606-g10684a45-1bionic [24.6 MB] 2020-11-29T15:53:24.229 INFO:teuthology.orchestra.run.smithi168.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7606-g10684a45-1bionic [3,808 B] 2020-11-29T15:53:24.229 INFO:teuthology.orchestra.run.smithi168.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7606-g10684a45-1bionic [561 kB] 2020-11-29T15:53:24.277 INFO:teuthology.orchestra.run.smithi168.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7606-g10684a45-1bionic [1,945 kB] 2020-11-29T15:53:24.407 INFO:teuthology.orchestra.run.smithi168.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7606-g10684a45-1bionic [51.7 kB] 2020-11-29T15:53:24.408 INFO:teuthology.orchestra.run.smithi168.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7606-g10684a45-1bionic [56.8 kB] 2020-11-29T15:53:24.411 INFO:teuthology.orchestra.run.smithi168.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7606-g10684a45-1bionic [2,878 kB] 2020-11-29T15:53:24.634 INFO:teuthology.orchestra.run.smithi168.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7606-g10684a45-1bionic [7,870 kB] 2020-11-29T15:53:24.694 INFO:teuthology.orchestra.run.smithi184.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7606-g10684a45-1bionic [3,808 B] 2020-11-29T15:53:24.695 INFO:teuthology.orchestra.run.smithi184.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7606-g10684a45-1bionic [561 kB] 2020-11-29T15:53:24.741 INFO:teuthology.orchestra.run.smithi184.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7606-g10684a45-1bionic [1,945 kB] 2020-11-29T15:53:24.889 INFO:teuthology.orchestra.run.smithi184.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7606-g10684a45-1bionic [51.7 kB] 2020-11-29T15:53:24.891 INFO:teuthology.orchestra.run.smithi184.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7606-g10684a45-1bionic [56.8 kB] 2020-11-29T15:53:24.919 INFO:teuthology.orchestra.run.smithi184.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7606-g10684a45-1bionic [2,878 kB] 2020-11-29T15:53:25.126 INFO:teuthology.orchestra.run.smithi184.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7606-g10684a45-1bionic [7,870 kB] 2020-11-29T15:53:25.174 INFO:teuthology.orchestra.run.smithi168.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7606-g10684a45-1bionic [3,928 B] 2020-11-29T15:53:25.175 INFO:teuthology.orchestra.run.smithi168.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7606-g10684a45-1bionic [22.3 kB] 2020-11-29T15:53:25.175 INFO:teuthology.orchestra.run.smithi168.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7606-g10684a45-1bionic [50.4 MB] 2020-11-29T15:53:25.736 INFO:teuthology.orchestra.run.smithi184.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7606-g10684a45-1bionic [3,928 B] 2020-11-29T15:53:25.736 INFO:teuthology.orchestra.run.smithi184.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7606-g10684a45-1bionic [22.3 kB] 2020-11-29T15:53:25.737 INFO:teuthology.orchestra.run.smithi184.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7606-g10684a45-1bionic [50.4 MB] 2020-11-29T15:53:28.913 INFO:teuthology.orchestra.run.smithi168.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7606-g10684a45-1bionic [19.5 kB] 2020-11-29T15:53:28.913 INFO:teuthology.orchestra.run.smithi168.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7606-g10684a45-1bionic [9,887 kB] 2020-11-29T15:53:29.617 INFO:teuthology.orchestra.run.smithi168.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7606-g10684a45-1bionic [77.6 kB] 2020-11-29T15:53:29.733 INFO:teuthology.orchestra.run.smithi184.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7606-g10684a45-1bionic [19.5 kB] 2020-11-29T15:53:29.734 INFO:teuthology.orchestra.run.smithi184.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7606-g10684a45-1bionic [9,887 kB] 2020-11-29T15:53:30.184 INFO:teuthology.orchestra.run.smithi184.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rgw-d3n-cache-upstreaming_i20/10684a45d9b8cdfc272d8721720e890aa44052c6/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7606-g10684a45-1bionic [77.6 kB] 2020-11-29T15:53:30.356 INFO:teuthology.orchestra.run.smithi168.stdout:Fetched 169 MB in 10s (16.8 MB/s) 2020-11-29T15:53:30.475 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-29T15:53:30.512 INFO:teuthology.orchestra.run.smithi168.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-29T15:53:30.514 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-29T15:53:30.548 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-29T15:53:30.876 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-29T15:53:30.888 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-29T15:53:30.895 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 169 MB in 11s (16.0 MB/s) 2020-11-29T15:53:30.968 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-29T15:53:31.040 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-29T15:53:31.079 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-29T15:53:31.081 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-29T15:53:31.130 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-29T15:53:31.301 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../002-librbd1_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:31.408 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-29T15:53:31.419 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-29T15:53:31.422 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking librbd1 (16.0.0-7606-g10684a45-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-11-29T15:53:31.500 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-29T15:53:31.840 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../002-librbd1_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:31.971 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librbd1 (16.0.0-7606-g10684a45-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-11-29T15:53:31.980 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../003-librados2_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:32.085 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking librados2 (16.0.0-7606-g10684a45-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-11-29T15:53:32.543 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../003-librados2_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:32.674 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librados2 (16.0.0-7606-g10684a45-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-11-29T15:53:32.739 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package libcephfs2. 2020-11-29T15:53:32.750 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:32.789 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking libcephfs2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:33.058 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-rados. 2020-11-29T15:53:33.069 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:33.100 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-rados (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:33.319 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libcephfs2. 2020-11-29T15:53:33.331 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:33.369 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libcephfs2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:33.436 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-29T15:53:33.449 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:33.486 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-ceph-argparse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:33.647 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rados. 2020-11-29T15:53:33.658 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../005-python3-rados_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:33.697 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rados (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:33.781 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-cephfs. 2020-11-29T15:53:33.793 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:33.831 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-cephfs (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:34.024 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-29T15:53:34.037 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:34.074 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-ceph-argparse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:34.125 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-29T15:53:34.138 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:34.167 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-ceph-common (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:34.335 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cephfs. 2020-11-29T15:53:34.346 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:34.377 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cephfs (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:34.412 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-prettytable. 2020-11-29T15:53:34.423 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-29T15:53:34.454 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-29T15:53:34.690 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-rbd. 2020-11-29T15:53:34.696 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-29T15:53:34.700 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:34.707 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:34.740 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-rbd (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:34.746 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-ceph-common (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:35.057 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-prettytable. 2020-11-29T15:53:35.070 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-11-29T15:53:35.107 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-29T15:53:35.117 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-29T15:53:35.128 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-29T15:53:35.168 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-29T15:53:35.335 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rbd. 2020-11-29T15:53:35.346 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:35.376 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rbd (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:35.429 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-29T15:53:35.441 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-29T15:53:35.471 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-29T15:53:35.721 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-29T15:53:35.732 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-29T15:53:35.770 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-29T15:53:35.799 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package librgw2. 2020-11-29T15:53:35.811 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../013-librgw2_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:35.874 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking librgw2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:36.140 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-29T15:53:36.152 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-29T15:53:36.190 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-29T15:53:36.434 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-rgw. 2020-11-29T15:53:36.446 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:36.493 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-rgw (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:36.526 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librgw2. 2020-11-29T15:53:36.537 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../013-librgw2_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:36.576 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librgw2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:36.754 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-29T15:53:36.767 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-29T15:53:36.795 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-29T15:53:37.065 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-29T15:53:37.077 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-29T15:53:37.115 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-29T15:53:37.120 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rgw. 2020-11-29T15:53:37.132 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:37.170 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rgw (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:37.477 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package liboath0. 2020-11-29T15:53:37.480 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-29T15:53:37.489 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-29T15:53:37.492 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-29T15:53:37.527 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-29T15:53:37.531 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-29T15:53:37.788 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package libradosstriper1. 2020-11-29T15:53:37.801 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:37.825 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-29T15:53:37.829 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking libradosstriper1 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:37.838 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-29T15:53:37.875 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-29T15:53:38.228 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liboath0. 2020-11-29T15:53:38.240 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-11-29T15:53:38.266 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-common. 2020-11-29T15:53:38.278 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-29T15:53:38.279 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:38.316 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-common (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:38.580 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libradosstriper1. 2020-11-29T15:53:38.592 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:38.656 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libradosstriper1 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:39.109 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-common. 2020-11-29T15:53:39.120 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../019-ceph-common_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:39.159 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-common (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:40.098 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-base. 2020-11-29T15:53:40.110 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:40.148 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-base (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:41.039 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-base. 2020-11-29T15:53:41.052 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../020-ceph-base_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:41.078 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-dateutil. 2020-11-29T15:53:41.089 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-29T15:53:41.098 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-base (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:41.128 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-29T15:53:41.389 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-29T15:53:41.400 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-29T15:53:41.431 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-29T15:53:41.667 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-29T15:53:41.679 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-29T15:53:41.717 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-29T15:53:42.011 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-dateutil. 2020-11-29T15:53:42.023 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-11-29T15:53:42.045 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-cryptography. 2020-11-29T15:53:42.052 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-29T15:53:42.056 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-29T15:53:42.095 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-29T15:53:42.457 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-openssl. 2020-11-29T15:53:42.470 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-29T15:53:42.532 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-29T15:53:42.671 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-29T15:53:42.683 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-29T15:53:42.721 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-29T15:53:42.991 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-29T15:53:43.003 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-29T15:53:43.009 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-29T15:53:43.021 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:43.041 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-29T15:53:43.059 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:43.361 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cryptography. 2020-11-29T15:53:43.372 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-29T15:53:43.406 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-29T15:53:43.410 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-29T15:53:43.417 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-29T15:53:43.456 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-29T15:53:43.750 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-29T15:53:43.762 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-29T15:53:43.780 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-openssl. 2020-11-29T15:53:43.792 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-29T15:53:43.800 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-29T15:53:43.830 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-29T15:53:44.091 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-29T15:53:44.102 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:44.137 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-jwt. 2020-11-29T15:53:44.140 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:44.149 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-29T15:53:44.187 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-29T15:53:44.494 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-logutils. 2020-11-29T15:53:44.505 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-29T15:53:44.520 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-29T15:53:44.533 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-29T15:53:44.553 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-29T15:53:44.570 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-29T15:53:44.839 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-29T15:53:44.851 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-29T15:53:44.872 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-29T15:53:44.884 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-29T15:53:44.889 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-29T15:53:44.922 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-29T15:53:45.158 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-mako. 2020-11-29T15:53:45.169 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-29T15:53:45.208 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-29T15:53:45.268 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jwt. 2020-11-29T15:53:45.279 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-29T15:53:45.318 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-29T15:53:45.495 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-29T15:53:45.507 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-29T15:53:45.537 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-29T15:53:45.606 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-logutils. 2020-11-29T15:53:45.617 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-11-29T15:53:45.656 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-29T15:53:45.848 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-29T15:53:45.861 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-29T15:53:45.898 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-29T15:53:45.959 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-29T15:53:45.971 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-29T15:53:46.017 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-29T15:53:46.192 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-webob. 2020-11-29T15:53:46.204 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-29T15:53:46.242 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-29T15:53:46.320 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-mako. 2020-11-29T15:53:46.330 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-29T15:53:46.370 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-29T15:53:46.570 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-bs4. 2020-11-29T15:53:46.583 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-29T15:53:46.620 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-29T15:53:46.689 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-29T15:53:46.700 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-29T15:53:46.739 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-29T15:53:46.923 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-waitress. 2020-11-29T15:53:46.936 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-29T15:53:46.965 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-29T15:53:47.000 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-29T15:53:47.012 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-29T15:53:47.050 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-29T15:53:47.252 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-tempita. 2020-11-29T15:53:47.264 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-29T15:53:47.293 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-29T15:53:47.386 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webob. 2020-11-29T15:53:47.399 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-29T15:53:47.436 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-29T15:53:47.596 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-paste. 2020-11-29T15:53:47.608 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-29T15:53:47.646 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-29T15:53:47.747 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-bs4. 2020-11-29T15:53:47.760 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-11-29T15:53:47.797 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-29T15:53:48.041 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-29T15:53:48.052 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-29T15:53:48.083 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-29T15:53:48.117 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-waitress. 2020-11-29T15:53:48.129 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-11-29T15:53:48.167 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-29T15:53:48.361 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-29T15:53:48.374 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-29T15:53:48.427 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-29T15:53:48.486 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tempita. 2020-11-29T15:53:48.499 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-11-29T15:53:48.536 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-29T15:53:48.724 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-webtest. 2020-11-29T15:53:48.735 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-29T15:53:48.774 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-29T15:53:48.822 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-paste. 2020-11-29T15:53:48.833 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-29T15:53:48.872 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-29T15:53:49.052 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pecan. 2020-11-29T15:53:49.062 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-29T15:53:49.093 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-29T15:53:49.175 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-29T15:53:49.188 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-29T15:53:49.225 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-29T15:53:49.440 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-29T15:53:49.453 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-29T15:53:49.490 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-29T15:53:49.528 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-29T15:53:49.540 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-29T15:53:49.578 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-29T15:53:49.827 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mgr. 2020-11-29T15:53:49.839 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:49.872 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webtest. 2020-11-29T15:53:49.877 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mgr (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:49.884 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-29T15:53:49.939 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-29T15:53:50.258 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pecan. 2020-11-29T15:53:50.271 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-11-29T15:53:50.308 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-29T15:53:50.322 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mon. 2020-11-29T15:53:50.335 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:50.380 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mon (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:50.629 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-29T15:53:50.640 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-29T15:53:50.679 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-29T15:53:51.041 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr. 2020-11-29T15:53:51.052 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:51.090 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:51.317 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-osd. 2020-11-29T15:53:51.328 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:51.367 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-osd (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:51.560 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mon. 2020-11-29T15:53:51.572 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:51.610 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mon (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:53.247 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-osd. 2020-11-29T15:53:53.261 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:53.297 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-osd (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:53.981 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph. 2020-11-29T15:53:53.993 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../048-ceph_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:54.031 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:54.350 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-fuse. 2020-11-29T15:53:54.362 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:54.400 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-fuse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:54.712 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mds. 2020-11-29T15:53:54.725 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:54.754 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mds (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:55.190 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package cephadm. 2020-11-29T15:53:55.202 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../051-cephadm_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:55.240 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking cephadm (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:55.310 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph. 2020-11-29T15:53:55.322 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../048-ceph_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:55.360 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:55.560 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-jinja2. 2020-11-29T15:53:55.573 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-29T15:53:55.610 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-29T15:53:55.662 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-fuse. 2020-11-29T15:53:55.675 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:55.712 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-fuse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:55.946 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-29T15:53:55.959 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:55.996 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:56.099 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mds. 2020-11-29T15:53:56.112 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:56.149 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mds (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:56.291 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-29T15:53:56.304 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-29T15:53:56.333 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-29T15:53:56.577 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-routes. 2020-11-29T15:53:56.589 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-29T15:53:56.619 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-29T15:53:56.635 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephadm. 2020-11-29T15:53:56.647 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../051-cephadm_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:53:56.685 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephadm (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:56.897 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-29T15:53:56.910 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:56.947 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:57.004 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jinja2. 2020-11-29T15:53:57.016 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-29T15:53:57.054 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-29T15:53:57.357 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-29T15:53:57.370 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:57.407 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:57.727 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-29T15:53:57.739 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-11-29T15:53:57.759 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-numpy. 2020-11-29T15:53:57.771 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-29T15:53:57.777 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-29T15:53:57.809 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-29T15:53:58.079 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-routes. 2020-11-29T15:53:58.092 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-11-29T15:53:58.129 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-29T15:53:58.365 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-decorator. 2020-11-29T15:53:58.377 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-29T15:53:58.414 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-29T15:53:58.423 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-29T15:53:58.437 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:53:58.473 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:53:58.726 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-scipy. 2020-11-29T15:53:58.738 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-29T15:53:58.776 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-29T15:53:59.374 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-numpy. 2020-11-29T15:53:59.386 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-29T15:53:59.424 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-29T15:53:59.963 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-decorator. 2020-11-29T15:53:59.968 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-29T15:53:59.975 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-11-29T15:53:59.981 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-29T15:54:00.013 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-29T15:54:00.018 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-29T15:54:00.324 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-scipy. 2020-11-29T15:54:00.336 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-29T15:54:00.374 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-29T15:54:00.480 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-joblib. 2020-11-29T15:54:00.493 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-29T15:54:00.530 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-29T15:54:00.859 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-sklearn. 2020-11-29T15:54:00.872 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-29T15:54:00.917 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-29T15:54:01.459 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-29T15:54:01.470 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:54:01.509 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:01.629 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-29T15:54:01.642 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-29T15:54:01.682 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-29T15:54:02.135 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-joblib. 2020-11-29T15:54:02.147 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-11-29T15:54:02.185 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-29T15:54:02.513 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sklearn. 2020-11-29T15:54:02.526 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-11-29T15:54:02.563 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-29T15:54:02.695 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-monotonic. 2020-11-29T15:54:02.701 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-29T15:54:02.745 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-29T15:54:03.039 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-fasteners. 2020-11-29T15:54:03.052 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-29T15:54:03.089 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-29T15:54:03.127 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-29T15:54:03.140 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:54:03.177 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:03.392 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-httplib2. 2020-11-29T15:54:03.403 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-29T15:54:03.442 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-29T15:54:03.720 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-29T15:54:03.732 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-29T15:54:03.769 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-29T15:54:04.039 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-29T15:54:04.052 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-29T15:54:04.080 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-29T15:54:04.297 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-monotonic. 2020-11-29T15:54:04.309 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-11-29T15:54:04.347 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-29T15:54:04.375 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-rsa. 2020-11-29T15:54:04.389 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-29T15:54:04.425 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-29T15:54:04.649 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-fasteners. 2020-11-29T15:54:04.661 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-11-29T15:54:04.699 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-29T15:54:04.737 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-29T15:54:04.749 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-29T15:54:04.787 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-29T15:54:04.994 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-httplib2. 2020-11-29T15:54:05.007 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-29T15:54:05.044 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-29T15:54:05.217 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-29T15:54:05.228 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-29T15:54:05.266 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-29T15:54:05.347 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-29T15:54:05.360 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-29T15:54:05.397 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-29T15:54:05.603 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-setuptools. 2020-11-29T15:54:05.616 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-29T15:54:05.661 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-29T15:54:05.692 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-29T15:54:05.704 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-29T15:54:05.733 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-29T15:54:06.006 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-websocket. 2020-11-29T15:54:06.011 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rsa. 2020-11-29T15:54:06.017 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-29T15:54:06.025 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-11-29T15:54:06.056 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-29T15:54:06.069 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-29T15:54:06.356 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-29T15:54:06.367 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-29T15:54:06.370 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-29T15:54:06.379 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-29T15:54:06.414 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-29T15:54:06.417 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-29T15:54:06.847 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-29T15:54:06.857 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:54:06.895 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-29T15:54:06.895 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:06.908 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-29T15:54:06.945 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-29T15:54:07.215 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-29T15:54:07.226 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-29T15:54:07.256 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-29T15:54:07.281 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-setuptools. 2020-11-29T15:54:07.293 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-11-29T15:54:07.331 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-29T15:54:07.618 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-29T15:54:07.630 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-29T15:54:07.670 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-29T15:54:07.709 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-websocket. 2020-11-29T15:54:07.723 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-29T15:54:07.768 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-29T15:54:07.970 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-29T15:54:07.982 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:54:08.020 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-mgr-rook (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:08.079 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-29T15:54:08.090 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-29T15:54:08.129 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-29T15:54:08.340 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-29T15:54:08.353 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-29T15:54:08.390 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-29T15:54:08.583 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-29T15:54:08.594 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:54:08.633 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:08.701 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-29T15:54:08.715 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-29T15:54:08.837 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-29T15:54:08.936 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-29T15:54:08.948 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-11-29T15:54:08.977 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-29T15:54:09.097 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package jq. 2020-11-29T15:54:09.110 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-29T15:54:09.172 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-29T15:54:09.314 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-29T15:54:09.328 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-29T15:54:09.375 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-29T15:54:09.433 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package socat. 2020-11-29T15:54:09.445 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-29T15:54:09.483 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-29T15:54:09.675 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-29T15:54:09.688 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-7606-g10684a45-1bionic_all.deb ... 2020-11-29T15:54:09.725 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-rook (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:09.828 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-29T15:54:09.841 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-29T15:54:09.878 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-29T15:54:10.046 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-29T15:54:10.059 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-11-29T15:54:10.104 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-29T15:54:10.122 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package xmlstarlet. 2020-11-29T15:54:10.136 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-29T15:54:10.180 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-29T15:54:10.440 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-29T15:54:10.452 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-29T15:54:10.510 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package ceph-test. 2020-11-29T15:54:10.522 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:54:10.559 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking ceph-test (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:10.575 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-29T15:54:10.846 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package jq. 2020-11-29T15:54:10.857 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-11-29T15:54:10.895 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-29T15:54:11.123 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package socat. 2020-11-29T15:54:11.135 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-29T15:54:11.164 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-29T15:54:11.551 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-29T15:54:11.564 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-29T15:54:11.601 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-29T15:54:11.879 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package xmlstarlet. 2020-11-29T15:54:11.892 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-29T15:54:11.929 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-29T15:54:12.250 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-test. 2020-11-29T15:54:12.262 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../085-ceph-test_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:54:12.299 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-test (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:14.196 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-29T15:54:14.208 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:54:14.246 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking libcephfs-dev (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:14.532 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package nvme-cli. 2020-11-29T15:54:14.545 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2020-11-29T15:54:14.574 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2020-11-29T15:54:14.928 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python-attr. 2020-11-29T15:54:14.941 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-29T15:54:14.978 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-29T15:54:15.272 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python-funcsigs. 2020-11-29T15:54:15.284 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-29T15:54:15.322 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-29T15:54:15.608 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python-pluggy. 2020-11-29T15:54:15.619 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-29T15:54:15.658 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-29T15:54:15.952 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python-py. 2020-11-29T15:54:15.964 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-29T15:54:15.978 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-29T15:54:15.992 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:54:15.994 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-29T15:54:16.028 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libcephfs-dev (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:16.247 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python-pytest. 2020-11-29T15:54:16.261 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-29T15:54:16.297 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package nvme-cli. 2020-11-29T15:54:16.305 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-29T15:54:16.309 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2020-11-29T15:54:16.348 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2020-11-29T15:54:16.609 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-attr. 2020-11-29T15:54:16.621 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-29T15:54:16.650 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-29T15:54:16.677 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-attr. 2020-11-29T15:54:16.689 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-11-29T15:54:16.727 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-29T15:54:16.987 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-cycler. 2020-11-29T15:54:17.001 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-29T15:54:17.030 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-funcsigs. 2020-11-29T15:54:17.044 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-11-29T15:54:17.045 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-29T15:54:17.088 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-29T15:54:17.347 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-webencodings. 2020-11-29T15:54:17.357 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-pluggy. 2020-11-29T15:54:17.359 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-29T15:54:17.372 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-11-29T15:54:17.397 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-29T15:54:17.407 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-29T15:54:17.702 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-py. 2020-11-29T15:54:17.708 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-html5lib. 2020-11-29T15:54:17.713 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-11-29T15:54:17.722 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-29T15:54:17.752 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-29T15:54:17.766 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-29T15:54:18.063 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-pytest. 2020-11-29T15:54:18.077 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-11-29T15:54:18.111 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-29T15:54:18.121 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-29T15:54:18.124 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-29T15:54:18.153 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-29T15:54:18.475 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-attr. 2020-11-29T15:54:18.488 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-11-29T15:54:18.525 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-29T15:54:18.564 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-29T15:54:18.571 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-29T15:54:18.614 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-29T15:54:18.828 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cycler. 2020-11-29T15:54:18.839 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-11-29T15:54:18.878 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-29T15:54:18.934 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-tz. 2020-11-29T15:54:18.946 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-29T15:54:18.984 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-29T15:54:19.164 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webencodings. 2020-11-29T15:54:19.178 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-11-29T15:54:19.222 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-29T15:54:19.303 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-29T15:54:19.316 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-29T15:54:19.353 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-29T15:54:19.500 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-html5lib. 2020-11-29T15:54:19.512 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-11-29T15:54:19.550 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-29T15:54:19.887 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-29T15:54:19.899 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-29T15:54:19.937 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-29T15:54:19.983 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-olefile. 2020-11-29T15:54:19.998 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-29T15:54:20.033 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-29T15:54:20.344 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pastescript. 2020-11-29T15:54:20.357 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-29T15:54:20.365 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-29T15:54:20.377 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-29T15:54:20.394 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-29T15:54:20.415 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-29T15:54:20.676 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tz. 2020-11-29T15:54:20.690 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-11-29T15:54:20.714 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-29T15:54:20.726 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-29T15:54:20.728 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-29T15:54:20.764 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-29T15:54:20.954 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-29T15:54:20.967 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-29T15:54:21.004 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-29T15:54:21.126 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pluggy. 2020-11-29T15:54:21.139 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-29T15:54:21.184 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-29T15:54:21.453 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-py. 2020-11-29T15:54:21.467 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-29T15:54:21.503 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-29T15:54:21.693 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-olefile. 2020-11-29T15:54:21.706 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-11-29T15:54:21.751 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-29T15:54:21.798 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-29T15:54:21.809 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-29T15:54:21.847 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-29T15:54:22.071 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pastescript. 2020-11-29T15:54:22.084 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-11-29T15:54:22.120 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-29T15:54:22.167 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-pytest. 2020-11-29T15:54:22.180 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-29T15:54:22.217 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-29T15:54:22.474 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-29T15:54:22.487 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-29T15:54:22.524 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-29T15:54:22.570 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-simplejson. 2020-11-29T15:54:22.584 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-29T15:54:22.628 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-29T15:54:22.869 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pluggy. 2020-11-29T15:54:22.882 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-11-29T15:54:22.935 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-29T15:54:22.948 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-29T15:54:22.962 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-29T15:54:23.006 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-29T15:54:23.238 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-py. 2020-11-29T15:54:23.250 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-11-29T15:54:23.288 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-29T15:54:23.443 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package radosgw. 2020-11-29T15:54:23.457 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../110-radosgw_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:54:23.501 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking radosgw (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:23.566 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-29T15:54:23.579 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-29T15:54:23.616 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-29T15:54:23.946 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pytest. 2020-11-29T15:54:23.960 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-11-29T15:54:24.002 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-29T15:54:24.331 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-simplejson. 2020-11-29T15:54:24.345 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-29T15:54:24.390 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-29T15:54:24.479 INFO:teuthology.orchestra.run.smithi168.stdout:Selecting previously unselected package rbd-fuse. 2020-11-29T15:54:24.492 INFO:teuthology.orchestra.run.smithi168.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:54:24.529 INFO:teuthology.orchestra.run.smithi168.stdout:Unpacking rbd-fuse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:24.717 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-29T15:54:24.732 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-29T15:54:24.767 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-29T15:54:24.823 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-29T15:54:25.135 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-29T15:54:25.196 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package radosgw. 2020-11-29T15:54:25.210 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../110-radosgw_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:54:25.254 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking radosgw (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:25.511 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-29T15:54:25.628 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-29T15:54:26.206 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-29T15:54:26.282 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package rbd-fuse. 2020-11-29T15:54:26.295 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-7606-g10684a45-1bionic_amd64.deb ... 2020-11-29T15:54:26.332 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking rbd-fuse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:26.582 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-29T15:54:26.609 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-29T15:54:26.861 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-29T15:54:26.921 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-29T15:54:27.151 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-29T15:54:27.281 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-29T15:54:27.352 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-29T15:54:27.407 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-29T15:54:27.462 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-29T15:54:27.772 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-29T15:54:27.934 INFO:teuthology.orchestra.run.smithi168.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-29T15:54:28.026 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-29T15:54:28.241 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-29T15:54:28.413 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-29T15:54:28.542 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-ceph-argparse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:28.665 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-29T15:54:28.844 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-29T15:54:28.949 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-29T15:54:29.142 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-29T15:54:29.144 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-29T15:54:29.254 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-29T15:54:29.268 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-29T15:54:29.611 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-29T15:54:29.792 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-29T15:54:30.098 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-29T15:54:30.165 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-29T15:54:30.391 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-ceph-argparse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:30.492 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-29T15:54:30.700 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-29T15:54:30.784 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python-py (1.5.2-1) ... 2020-11-29T15:54:31.001 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-29T15:54:31.010 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-29T15:54:31.127 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-29T15:54:31.337 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-29T15:54:31.409 INFO:teuthology.orchestra.run.smithi168.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-29T15:54:31.663 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-29T15:54:31.789 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-29T15:54:31.990 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-29T15:54:32.056 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-29T15:54:32.308 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-29T15:54:32.348 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-29T15:54:32.617 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-29T15:54:32.649 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-py (1.5.2-1) ... 2020-11-29T15:54:32.876 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-29T15:54:32.902 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-29T15:54:33.202 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-29T15:54:33.253 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-29T15:54:33.292 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-29T15:54:33.563 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-29T15:54:33.563 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-29T15:54:33.688 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-29T15:54:33.864 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up cephadm (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:33.865 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-29T15:54:34.149 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-29T15:54:34.467 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-29T15:54:34.606 INFO:teuthology.orchestra.run.smithi168.stdout:Adding system user cephadm....done 2020-11-29T15:54:34.754 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-29T15:54:34.760 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-29T15:54:35.021 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-29T15:54:35.111 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-29T15:54:35.147 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2020-11-29T15:54:35.282 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-29T15:54:35.454 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-29T15:54:35.642 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-29T15:54:35.747 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephadm (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:35.943 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-29T15:54:36.069 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-29T15:54:36.381 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user cephadm....done 2020-11-29T15:54:36.412 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-ceph-common (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:36.545 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-29T15:54:36.713 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-29T15:54:36.839 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-29T15:54:36.948 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2020-11-29T15:54:37.006 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-29T15:54:37.074 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-29T15:54:37.140 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-29T15:54:37.409 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-29T15:54:37.483 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-29T15:54:37.693 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-29T15:54:37.819 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-29T15:54:37.834 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-29T15:54:37.924 INFO:teuthology.orchestra.run.smithi168.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-29T15:54:37.957 INFO:teuthology.orchestra.run.smithi168.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-29T15:54:38.155 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-ceph-common (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:38.195 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-29T15:54:38.464 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-29T15:54:38.498 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-29T15:54:38.740 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-29T15:54:38.780 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-29T15:54:38.866 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-29T15:54:39.123 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-29T15:54:39.217 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-29T15:54:39.426 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-29T15:54:39.552 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-29T15:54:39.619 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-29T15:54:39.642 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-29T15:54:39.674 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-29T15:54:39.921 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-29T15:54:40.021 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-29T15:54:40.094 INFO:teuthology.orchestra.run.smithi168.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-29T15:54:40.222 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-29T15:54:40.339 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-29T15:54:40.498 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-29T15:54:40.532 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-29T15:54:40.658 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-29T15:54:40.858 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-29T15:54:40.910 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-29T15:54:41.035 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-29T15:54:41.159 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-29T15:54:41.352 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-29T15:54:41.437 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-29T15:54:41.538 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-29T15:54:41.762 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-29T15:54:41.839 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-29T15:54:41.859 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-29T15:54:42.106 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-29T15:54:42.157 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-29T15:54:42.265 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-29T15:54:42.392 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-29T15:54:42.459 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-29T15:54:42.643 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-29T15:54:42.759 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-29T15:54:42.786 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-29T15:54:43.094 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-29T15:54:43.187 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-29T15:54:43.313 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-29T15:54:43.361 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-29T15:54:43.606 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-29T15:54:43.646 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-29T15:54:43.907 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-29T15:54:43.930 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-29T15:54:44.208 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-29T15:54:44.526 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-29T15:54:44.532 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-29T15:54:44.833 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-29T15:54:44.877 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-29T15:54:44.959 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-29T15:54:45.204 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-29T15:54:45.513 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-29T15:54:45.840 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-29T15:54:46.097 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-29T15:54:46.191 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-29T15:54:46.407 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-29T15:54:46.509 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-29T15:54:46.636 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-29T15:54:46.725 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-29T15:54:46.843 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-29T15:54:47.194 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-29T15:54:47.429 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-29T15:54:47.530 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-29T15:54:47.656 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up librados2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:47.782 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-29T15:54:47.807 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-29T15:54:48.100 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up libcephfs2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:48.117 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-29T15:54:48.234 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up libcephfs-dev (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:48.335 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-rados (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:48.435 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-29T15:54:48.436 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-29T15:54:48.561 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-29T15:54:48.729 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-29T15:54:49.013 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-29T15:54:49.079 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-29T15:54:49.339 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-29T15:54:49.348 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-29T15:54:49.504 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-29T15:54:49.667 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librados2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:49.793 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-29T15:54:50.102 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libcephfs2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:50.228 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libcephfs-dev (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:50.277 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-29T15:54:50.354 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rados (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:50.480 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-29T15:54:50.790 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-29T15:54:51.073 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-29T15:54:51.091 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-29T15:54:51.366 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up libradosstriper1 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:51.409 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-29T15:54:51.492 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-cephfs (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:51.777 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up librgw2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:51.878 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-fuse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:52.081 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-29T15:54:52.082 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-29T15:54:52.354 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up librbd1 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:52.356 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-29T15:54:52.472 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-29T15:54:52.553 INFO:teuthology.orchestra.run.smithi168.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-29T15:54:52.593 INFO:teuthology.orchestra.run.smithi168.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-29T15:54:52.892 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-29T15:54:53.144 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-29T15:54:53.420 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libradosstriper1 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:53.538 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cephfs (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:53.553 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mgr-modules-core (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:53.653 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-29T15:54:53.814 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librgw2 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:53.947 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-rgw (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:53.948 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-fuse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:54.065 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up rbd-fuse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:54.148 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-29T15:54:54.149 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-29T15:54:54.191 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up python3-rbd (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:54.317 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-common (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:54.442 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librbd1 (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:54.568 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-29T15:54:54.664 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-29T15:54:54.696 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-29T15:54:54.768 INFO:teuthology.orchestra.run.smithi168.stdout:Adding group ceph....done 2020-11-29T15:54:54.997 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-29T15:54:55.154 INFO:teuthology.orchestra.run.smithi168.stdout:Adding system user ceph....done 2020-11-29T15:54:55.346 INFO:teuthology.orchestra.run.smithi168.stdout:Setting system user ceph properties....done 2020-11-29T15:54:55.501 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-29T15:54:55.641 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-modules-core (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:55.714 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-29T15:54:55.742 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-29T15:54:55.994 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up radosgw (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:56.060 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rgw (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:56.186 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up rbd-fuse (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:56.236 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-29T15:54:56.236 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-29T15:54:56.321 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rbd (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:56.421 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-common (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:56.612 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-test (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:56.714 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-base (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:56.881 INFO:teuthology.orchestra.run.smithi184.stdout:Adding group ceph....done 2020-11-29T15:54:57.034 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-29T15:54:57.276 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user ceph....done 2020-11-29T15:54:57.315 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mgr (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:57.443 INFO:teuthology.orchestra.run.smithi184.stdout:Setting system user ceph properties....done 2020-11-29T15:54:57.531 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-29T15:54:57.531 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-29T15:54:57.587 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-29T15:54:57.809 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:57.822 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-29T15:54:57.951 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-osd (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:58.149 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up radosgw (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:58.384 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-29T15:54:58.384 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-29T15:54:58.410 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-29T15:54:58.410 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-29T15:54:58.637 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mgr-cephadm (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:58.755 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mgr-rook (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:58.826 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-test (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:58.872 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mds (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:58.952 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-base (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:59.083 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-29T15:54:59.083 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-29T15:54:59.357 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mon (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:59.459 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-29T15:54:59.576 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-29T15:54:59.577 INFO:teuthology.orchestra.run.smithi168.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-29T15:54:59.747 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:59.843 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:54:59.960 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-29T15:54:59.961 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-29T15:54:59.968 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph-mgr-dashboard (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:00.095 INFO:teuthology.orchestra.run.smithi168.stdout:Setting up ceph (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:00.196 INFO:teuthology.orchestra.run.smithi168.stdout:Processing triggers for systemd (237-3ubuntu10.43) ... 2020-11-29T15:55:00.248 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:00.363 INFO:teuthology.orchestra.run.smithi168.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-29T15:55:00.366 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-osd (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:00.799 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-29T15:55:00.799 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-29T15:55:01.077 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-cephadm (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:01.219 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-rook (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:01.362 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mds (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:01.577 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-29T15:55:01.578 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-29T15:55:01.855 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mon (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:02.057 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-29T15:55:02.057 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-29T15:55:02.331 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:02.432 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-dashboard (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:02.458 INFO:teuthology.orchestra.run.smithi168.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-29T15:55:02.500 INFO:teuthology.orchestra.run.smithi168.stdout:ureadahead will be reprofiled on next reboot 2020-11-29T15:55:02.550 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph (16.0.0-7606-g10684a45-1bionic) ... 2020-11-29T15:55:02.558 INFO:teuthology.orchestra.run.smithi168.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-29T15:55:02.677 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for systemd (237-3ubuntu10.43) ... 2020-11-29T15:55:02.869 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-29T15:55:03.022 INFO:teuthology.orchestra.run.smithi168.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-29T15:55:03.024 DEBUG:teuthology.parallel:result is None 2020-11-29T15:55:04.873 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-29T15:55:04.907 INFO:teuthology.orchestra.run.smithi184.stdout:ureadahead will be reprofiled on next reboot 2020-11-29T15:55:04.957 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-29T15:55:05.401 INFO:teuthology.orchestra.run.smithi184.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-29T15:55:05.404 DEBUG:teuthology.parallel:result is None 2020-11-29T15:55:05.405 INFO:teuthology.orchestra.run.smithi168:> sudo lsb_release -is 2020-11-29T15:55:05.454 INFO:teuthology.orchestra.run.smithi168.stdout:Ubuntu 2020-11-29T15:55:05.459 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-29T15:55:05.459 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=10684a45d9b8cdfc272d8721720e890aa44052c6 2020-11-29T15:55:05.627 INFO:teuthology.orchestra.run.smithi168:> dpkg-query -W -f '${Version}' ceph 2020-11-29T15:55:05.651 INFO:teuthology.orchestra.run.smithi168.stdout:16.0.0-7606-g10684a45-1bionic 2020-11-29T15:55:05.651 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7606-g10684a45-1bionic 2020-11-29T15:55:05.652 INFO:teuthology.task.install:The correct ceph version 16.0.0-7606-g10684a45-1bionic is installed. 2020-11-29T15:55:05.652 INFO:teuthology.orchestra.run.smithi184:> sudo lsb_release -is 2020-11-29T15:55:05.707 INFO:teuthology.orchestra.run.smithi184.stdout:Ubuntu 2020-11-29T15:55:05.712 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-29T15:55:05.713 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=10684a45d9b8cdfc272d8721720e890aa44052c6 2020-11-29T15:55:05.880 INFO:teuthology.orchestra.run.smithi184:> dpkg-query -W -f '${Version}' ceph 2020-11-29T15:55:05.905 INFO:teuthology.orchestra.run.smithi184.stdout:16.0.0-7606-g10684a45-1bionic 2020-11-29T15:55:05.906 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7606-g10684a45-1bionic 2020-11-29T15:55:05.906 INFO:teuthology.task.install:The correct ceph version 16.0.0-7606-g10684a45-1bionic is installed. 2020-11-29T15:55:05.906 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-29T15:55:05.907 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:05.907 INFO:teuthology.orchestra.run.smithi168:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-29T15:55:05.912 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-29T15:55:05.922 INFO:teuthology.orchestra.run.smithi168.stderr:24+1 records in 2020-11-29T15:55:05.922 INFO:teuthology.orchestra.run.smithi168.stderr:24+1 records out 2020-11-29T15:55:05.922 INFO:teuthology.orchestra.run.smithi168.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000225247 s, 56.2 MB/s 2020-11-29T15:55:05.923 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:05.923 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-29T15:55:05.952 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-29T15:55:05.961 INFO:teuthology.orchestra.run.smithi184.stderr:24+1 records in 2020-11-29T15:55:05.961 INFO:teuthology.orchestra.run.smithi184.stderr:24+1 records out 2020-11-29T15:55:05.962 INFO:teuthology.orchestra.run.smithi184.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000225334 s, 56.2 MB/s 2020-11-29T15:55:05.963 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-29T15:55:05.963 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:05.963 INFO:teuthology.orchestra.run.smithi168:> sudo dd of=/usr/bin/daemon-helper 2020-11-29T15:55:05.968 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-29T15:55:05.976 INFO:teuthology.orchestra.run.smithi168.stderr:5+1 records in 2020-11-29T15:55:05.977 INFO:teuthology.orchestra.run.smithi168.stderr:5+1 records out 2020-11-29T15:55:05.977 INFO:teuthology.orchestra.run.smithi168.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.3536e-05 s, 32.7 MB/s 2020-11-29T15:55:05.978 INFO:teuthology.orchestra.run.smithi168:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-29T15:55:06.035 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:06.035 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2020-11-29T15:55:06.040 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-29T15:55:06.049 INFO:teuthology.orchestra.run.smithi184.stderr:5+1 records in 2020-11-29T15:55:06.049 INFO:teuthology.orchestra.run.smithi184.stderr:5+1 records out 2020-11-29T15:55:06.049 INFO:teuthology.orchestra.run.smithi184.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.4093e-05 s, 32.5 MB/s 2020-11-29T15:55:06.050 INFO:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-29T15:55:06.110 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-29T15:55:06.110 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:06.111 INFO:teuthology.orchestra.run.smithi168:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-29T15:55:06.115 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-29T15:55:06.124 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:06.125 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:06.125 INFO:teuthology.orchestra.run.smithi168.stderr:299 bytes copied, 6.9116e-05 s, 4.3 MB/s 2020-11-29T15:55:06.126 INFO:teuthology.orchestra.run.smithi168:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-29T15:55:06.187 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:06.187 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-29T15:55:06.192 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-29T15:55:06.201 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:06.202 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:06.202 INFO:teuthology.orchestra.run.smithi184.stderr:299 bytes copied, 6.834e-05 s, 4.4 MB/s 2020-11-29T15:55:06.203 INFO:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-29T15:55:06.257 INFO:teuthology.run_tasks:Running task ceph... 2020-11-29T15:55:06.312 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-29T15:55:06.313 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 777 /var/log/ceph 2020-11-29T15:55:06.316 INFO:teuthology.orchestra.run.smithi184:> sudo chmod 777 /var/log/ceph 2020-11-29T15:55:06.329 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-29T15:55:06.330 INFO:teuthology.orchestra.run.smithi168:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-29T15:55:06.374 INFO:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-29T15:55:06.389 INFO:tasks.ceph:Creating extra log directories... 2020-11-29T15:55:06.389 INFO:teuthology.orchestra.run.smithi168:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-29T15:55:06.430 INFO:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-29T15:55:06.540 INFO:tasks.ceph:Creating ceph cluster c1... 2020-11-29T15:55:06.540 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-29T15:55:06.540 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi/5665427', 'branch': 'wip-rgw-d3n-cache-upstreaming_i20', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'mkogan@redhat.com', 'first_in_suite': False, 'job_id': '5665427', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-rgw-d3n-cache-upstreaming_i20'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-rgw-d3n-cache-upstreaming_i20', 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6'}}, 'owner': 'scheduled_mkogan@teuthology', 'priority': 150, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-d3n-cache-upstreaming_i20', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-d3n-cache-upstreaming_i20/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-29_15:41:09', 'tube': 'smithi', 'user': 'mkogan', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.25223', 'targets': {'smithi168.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdrYZbb8MPSUJwpZ8Q7VVCJhkVlWIs7odbYMnfk3AU5J446G411sRshkndZuW7PE3fXPGJD8BInXKRBhBSy+nkRiv9n/om6poKcIG5Yv1SdpNdOtxTpOXzBjpOeYYLVHrUpq/cza4Nh+IBhyhavgDiU5L0pP215lQHswHSTZPAkTMzNuWQQ2QhjV+d9kzDSfggdzVyU9Xj4q3SE+MQU6Aptm3J/Qhrxj11C8hUB2LYJGFFqKw8lWW50ylVaCByHuuLLmQhHRhl5Ysgpe5Rnnpk7GF10RN3yvQxmeW6XEl1ZrKGdk232VZWx7BGHW3+VF8DiLW6t0umenxaKPyTTX8V', 'smithi184.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDadENPl2I5HwceuElxWd2qNVQqbzVzle1/SaPOx61ShU45ELWegCqyHgUkO4iCRopi05bspLcoF0hyQ7DncvrWAcmlw1mimGxjM+6UgakidoKNBba9EZ5FTG2O9nEXoNFYIe5DXoA+/QcrShkDQnTKWuKsb41jXhS1UrPNCK+wEZ98w5ClbEL8YMMMFdH6LNGyizsXvmFWKK4VLPhWlqGVa2zi+P8P28XoUn3fI8OvQ8DTnlDhRZAbuduRfopcjIKERfn2SU2z+0dmj8bUoTL5vS809ClPLFR2xTprUkI/dmRJE4JGUb+E/ns79IIn+AG2FCys0qZ0YGaLrmTzPPXT'}} 2020-11-29T15:55:06.541 INFO:teuthology.orchestra.run.smithi168:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-29T15:55:06.544 INFO:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-11-29T15:55:06.551 INFO:teuthology.orchestra.run.smithi168:> sudo install -d -m0777 -- /var/run/ceph 2020-11-29T15:55:06.590 INFO:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2020-11-29T15:55:06.606 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:06.606 INFO:teuthology.orchestra.run.smithi168:> dd if=/scratch_devs of=/dev/stdout 2020-11-29T15:55:06.649 INFO:teuthology.orchestra.run.smithi168.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-29T15:55:06.651 INFO:teuthology.orchestra.run.smithi168.stdout:/dev/vg_nvme/lv_1 2020-11-29T15:55:06.651 INFO:teuthology.orchestra.run.smithi168.stdout:/dev/vg_nvme/lv_2 2020-11-29T15:55:06.652 INFO:teuthology.orchestra.run.smithi168.stdout:/dev/vg_nvme/lv_3 2020-11-29T15:55:06.652 INFO:teuthology.orchestra.run.smithi168.stdout:/dev/vg_nvme/lv_4 2020-11-29T15:55:06.652 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:06.652 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:06.653 INFO:teuthology.orchestra.run.smithi168.stderr:72 bytes copied, 0.000146284 s, 492 kB/s 2020-11-29T15:55:06.653 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-29T15:55:06.654 INFO:teuthology.orchestra.run.smithi168:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-29T15:55:06.699 INFO:teuthology.orchestra.run.smithi168.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-29T15:55:06.700 INFO:teuthology.orchestra.run.smithi168.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-29T15:55:06.700 INFO:teuthology.orchestra.run.smithi168.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-11-29T15:55:06.700 INFO:teuthology.orchestra.run.smithi168.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-29T15:55:06.700 INFO:teuthology.orchestra.run.smithi168.stdout:Access: 2020-11-29 15:52:46.428502803 +0000 2020-11-29T15:55:06.700 INFO:teuthology.orchestra.run.smithi168.stdout:Modify: 2020-11-29 15:52:46.340458804 +0000 2020-11-29T15:55:06.701 INFO:teuthology.orchestra.run.smithi168.stdout:Change: 2020-11-29 15:52:46.340458804 +0000 2020-11-29T15:55:06.701 INFO:teuthology.orchestra.run.smithi168.stdout: Birth: - 2020-11-29T15:55:06.709 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records in 2020-11-29T15:55:06.709 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records out 2020-11-29T15:55:06.709 INFO:teuthology.orchestra.run.smithi168.stderr:512 bytes copied, 0.000318392 s, 1.6 MB/s 2020-11-29T15:55:06.714 INFO:teuthology.orchestra.run.smithi168:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-29T15:55:06.763 INFO:teuthology.orchestra.run.smithi168.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-29T15:55:06.764 INFO:teuthology.orchestra.run.smithi168.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-29T15:55:06.764 INFO:teuthology.orchestra.run.smithi168.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-11-29T15:55:06.764 INFO:teuthology.orchestra.run.smithi168.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-29T15:55:06.764 INFO:teuthology.orchestra.run.smithi168.stdout:Access: 2020-11-29 15:52:46.668622799 +0000 2020-11-29T15:55:06.764 INFO:teuthology.orchestra.run.smithi168.stdout:Modify: 2020-11-29 15:52:46.564570801 +0000 2020-11-29T15:55:06.765 INFO:teuthology.orchestra.run.smithi168.stdout:Change: 2020-11-29 15:52:46.564570801 +0000 2020-11-29T15:55:06.765 INFO:teuthology.orchestra.run.smithi168.stdout: Birth: - 2020-11-29T15:55:06.772 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records in 2020-11-29T15:55:06.773 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records out 2020-11-29T15:55:06.773 INFO:teuthology.orchestra.run.smithi168.stderr:512 bytes copied, 0.000349186 s, 1.5 MB/s 2020-11-29T15:55:06.777 INFO:teuthology.orchestra.run.smithi168:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-29T15:55:06.827 INFO:teuthology.orchestra.run.smithi168.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-29T15:55:06.828 INFO:teuthology.orchestra.run.smithi168.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-29T15:55:06.828 INFO:teuthology.orchestra.run.smithi168.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-11-29T15:55:06.829 INFO:teuthology.orchestra.run.smithi168.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-29T15:55:06.829 INFO:teuthology.orchestra.run.smithi168.stdout:Access: 2020-11-29 15:52:46.896736797 +0000 2020-11-29T15:55:06.829 INFO:teuthology.orchestra.run.smithi168.stdout:Modify: 2020-11-29 15:52:46.800688798 +0000 2020-11-29T15:55:06.829 INFO:teuthology.orchestra.run.smithi168.stdout:Change: 2020-11-29 15:52:46.800688798 +0000 2020-11-29T15:55:06.829 INFO:teuthology.orchestra.run.smithi168.stdout: Birth: - 2020-11-29T15:55:06.836 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records in 2020-11-29T15:55:06.836 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records out 2020-11-29T15:55:06.836 INFO:teuthology.orchestra.run.smithi168.stderr:512 bytes copied, 0.000324084 s, 1.6 MB/s 2020-11-29T15:55:06.840 INFO:teuthology.orchestra.run.smithi168:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-29T15:55:06.887 INFO:teuthology.orchestra.run.smithi168.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-29T15:55:06.888 INFO:teuthology.orchestra.run.smithi168.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-29T15:55:06.888 INFO:teuthology.orchestra.run.smithi168.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-11-29T15:55:06.889 INFO:teuthology.orchestra.run.smithi168.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-29T15:55:06.889 INFO:teuthology.orchestra.run.smithi168.stdout:Access: 2020-11-29 15:52:47.132854792 +0000 2020-11-29T15:55:06.889 INFO:teuthology.orchestra.run.smithi168.stdout:Modify: 2020-11-29 15:52:47.036806793 +0000 2020-11-29T15:55:06.890 INFO:teuthology.orchestra.run.smithi168.stdout:Change: 2020-11-29 15:52:47.036806793 +0000 2020-11-29T15:55:06.890 INFO:teuthology.orchestra.run.smithi168.stdout: Birth: - 2020-11-29T15:55:06.897 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records in 2020-11-29T15:55:06.901 INFO:teuthology.orchestra.run.smithi168.stderr:1+0 records out 2020-11-29T15:55:06.901 INFO:teuthology.orchestra.run.smithi168.stderr:512 bytes copied, 0.000343213 s, 1.5 MB/s 2020-11-29T15:55:06.902 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-29T15:55:06.902 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi168.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-29T15:55:06.902 INFO:tasks.ceph:Generating config... 2020-11-29T15:55:06.904 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-29T15:55:06.904 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-29T15:55:06.904 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-29T15:55:06.905 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-29T15:55:06.905 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-29T15:55:06.905 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-29T15:55:06.905 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-29T15:55:06.906 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-29T15:55:06.906 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-29T15:55:06.906 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-29T15:55:06.907 INFO:tasks.ceph:[client] setuser = ceph 2020-11-29T15:55:06.907 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-29T15:55:06.907 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-29T15:55:06.907 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-29T15:55:06.908 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-29T15:55:06.908 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-29T15:55:06.908 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-29T15:55:06.908 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-29T15:55:06.909 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-29T15:55:06.909 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-29T15:55:06.909 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-29T15:55:06.910 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-29T15:55:06.910 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-29T15:55:06.910 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-29T15:55:06.910 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-29T15:55:06.911 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-29T15:55:06.911 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-29T15:55:06.911 INFO:tasks.ceph:Setting up c1.mon.a... 2020-11-29T15:55:06.912 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-11-29T15:55:06.975 INFO:teuthology.orchestra.run.smithi168.stdout:creating /etc/ceph/c1.keyring 2020-11-29T15:55:06.978 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-11-29T15:55:07.056 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-29T15:55:07.115 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.168')] 2020-11-29T15:55:07.116 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.168', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-29T15:55:07.118 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:07.118 INFO:teuthology.orchestra.run.smithi168:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-29T15:55:07.161 INFO:teuthology.orchestra.run.smithi168.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-29T15:55:07.162 INFO:teuthology.orchestra.run.smithi168.stderr:6+1 records in 2020-11-29T15:55:07.163 INFO:teuthology.orchestra.run.smithi168.stderr:6+1 records out 2020-11-29T15:55:07.163 INFO:teuthology.orchestra.run.smithi168.stderr:3136 bytes (3.1 kB, 3.1 KiB) copied, 0.000110961 s, 28.3 MB/s 2020-11-29T15:55:07.163 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.168 --print /home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:07.234 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:55:07.234 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:55:07.235 INFO:teuthology.orchestra.run.smithi168.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:07.235 INFO:teuthology.orchestra.run.smithi168.stdout:monmaptool: generated fsid 73679a84-1a68-4bdd-a7a1-215c2cc5bd23 2020-11-29T15:55:07.235 INFO:teuthology.orchestra.run.smithi168.stdout:epoch 0 2020-11-29T15:55:07.235 INFO:teuthology.orchestra.run.smithi168.stdout:fsid 73679a84-1a68-4bdd-a7a1-215c2cc5bd23 2020-11-29T15:55:07.235 INFO:teuthology.orchestra.run.smithi168.stdout:last_changed 2020-11-29T15:55:07.238788+0000 2020-11-29T15:55:07.236 INFO:teuthology.orchestra.run.smithi168.stdout:created 2020-11-29T15:55:07.238788+0000 2020-11-29T15:55:07.236 INFO:teuthology.orchestra.run.smithi168.stdout:min_mon_release 0 (unknown) 2020-11-29T15:55:07.236 INFO:teuthology.orchestra.run.smithi168.stdout:election_strategy: 1 2020-11-29T15:55:07.236 INFO:teuthology.orchestra.run.smithi168.stdout:0: [v2:172.21.15.168:3300/0,v1:172.21.15.168:6789/0] mon.a 2020-11-29T15:55:07.237 INFO:teuthology.orchestra.run.smithi168.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-11-29T15:55:07.238 INFO:teuthology.orchestra.run.smithi168:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-29T15:55:07.286 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 73679a84-1a68-4bdd-a7a1-215c2cc5bd23... 2020-11-29T15:55:07.288 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-29T15:55:07.330 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-11-29T15:55:07.361 INFO:teuthology.orchestra.run.smithi168.stdout:[global] 2020-11-29T15:55:07.361 INFO:teuthology.orchestra.run.smithi168.stdout: chdir = "" 2020-11-29T15:55:07.361 INFO:teuthology.orchestra.run.smithi168.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-29T15:55:07.362 INFO:teuthology.orchestra.run.smithi168.stdout: auth supported = cephx 2020-11-29T15:55:07.362 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.362 INFO:teuthology.orchestra.run.smithi168.stdout: filestore xattr use omap = true 2020-11-29T15:55:07.362 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.362 INFO:teuthology.orchestra.run.smithi168.stdout: mon clock drift allowed = 1.000 2020-11-29T15:55:07.363 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.363 INFO:teuthology.orchestra.run.smithi168.stdout: osd crush chooseleaf type = 0 2020-11-29T15:55:07.363 INFO:teuthology.orchestra.run.smithi168.stdout: auth debug = true 2020-11-29T15:55:07.363 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.363 INFO:teuthology.orchestra.run.smithi168.stdout: ms die on old message = true 2020-11-29T15:55:07.364 INFO:teuthology.orchestra.run.smithi168.stdout: ms die on bug = true 2020-11-29T15:55:07.364 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.364 INFO:teuthology.orchestra.run.smithi168.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-29T15:55:07.364 INFO:teuthology.orchestra.run.smithi168.stdout: mon pg warn max object skew = 0 2020-11-29T15:55:07.365 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.365 INFO:teuthology.orchestra.run.smithi168.stdout: # disable pg_autoscaler by default for new pools 2020-11-29T15:55:07.365 INFO:teuthology.orchestra.run.smithi168.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-29T15:55:07.365 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.365 INFO:teuthology.orchestra.run.smithi168.stdout: osd pool default size = 2 2020-11-29T15:55:07.366 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.366 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd allow primary affinity = true 2020-11-29T15:55:07.366 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd allow pg remap = true 2020-11-29T15:55:07.366 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on legacy crush tunables = false 2020-11-29T15:55:07.366 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on crush straw calc version zero = false 2020-11-29T15:55:07.367 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on no sortbitwise = false 2020-11-29T15:55:07.367 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on osd down out interval zero = false 2020-11-29T15:55:07.367 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on too few osds = false 2020-11-29T15:55:07.367 INFO:teuthology.orchestra.run.smithi168.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-29T15:55:07.368 INFO:teuthology.orchestra.run.smithi168.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-29T15:55:07.368 INFO:teuthology.orchestra.run.smithi168.stdout: mon_allow_pool_size_one = true 2020-11-29T15:55:07.368 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.368 INFO:teuthology.orchestra.run.smithi168.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-29T15:55:07.368 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.369 INFO:teuthology.orchestra.run.smithi168.stdout: osd default data pool replay window = 5 2020-11-29T15:55:07.369 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.369 INFO:teuthology.orchestra.run.smithi168.stdout: mon allow pool delete = true 2020-11-29T15:55:07.369 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.369 INFO:teuthology.orchestra.run.smithi168.stdout: mon cluster log file level = debug 2020-11-29T15:55:07.370 INFO:teuthology.orchestra.run.smithi168.stdout: debug asserts on shutdown = true 2020-11-29T15:55:07.370 INFO:teuthology.orchestra.run.smithi168.stdout: mon health detail to clog = false 2020-11-29T15:55:07.370 INFO:teuthology.orchestra.run.smithi168.stdout: mon host = 172.21.15.168 2020-11-29T15:55:07.370 INFO:teuthology.orchestra.run.smithi168.stdout: osd heartbeat grace = 40 2020-11-29T15:55:07.370 INFO:teuthology.orchestra.run.smithi168.stdout: fsid = 73679a84-1a68-4bdd-a7a1-215c2cc5bd23 2020-11-29T15:55:07.371 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.371 INFO:teuthology.orchestra.run.smithi168.stdout:[osd] 2020-11-29T15:55:07.371 INFO:teuthology.orchestra.run.smithi168.stdout: osd journal size = 100 2020-11-29T15:55:07.371 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.371 INFO:teuthology.orchestra.run.smithi168.stdout: osd scrub load threshold = 5.0 2020-11-29T15:55:07.372 INFO:teuthology.orchestra.run.smithi168.stdout: osd scrub max interval = 600 2020-11-29T15:55:07.372 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.372 INFO:teuthology.orchestra.run.smithi168.stdout: osd recover clone overlap = true 2020-11-29T15:55:07.372 INFO:teuthology.orchestra.run.smithi168.stdout: osd recovery max chunk = 1048576 2020-11-29T15:55:07.372 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.373 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug shutdown = true 2020-11-29T15:55:07.373 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug op order = true 2020-11-29T15:55:07.373 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug verify stray on activate = true 2020-11-29T15:55:07.373 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.373 INFO:teuthology.orchestra.run.smithi168.stdout: osd open classes on start = true 2020-11-29T15:55:07.374 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug pg log writeout = true 2020-11-29T15:55:07.374 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.374 INFO:teuthology.orchestra.run.smithi168.stdout: osd deep scrub update digest min age = 30 2020-11-29T15:55:07.374 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.374 INFO:teuthology.orchestra.run.smithi168.stdout: osd map max advance = 10 2020-11-29T15:55:07.375 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.375 INFO:teuthology.orchestra.run.smithi168.stdout: journal zero on create = true 2020-11-29T15:55:07.375 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.375 INFO:teuthology.orchestra.run.smithi168.stdout: filestore ondisk finisher threads = 3 2020-11-29T15:55:07.376 INFO:teuthology.orchestra.run.smithi168.stdout: filestore apply finisher threads = 3 2020-11-29T15:55:07.376 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.376 INFO:teuthology.orchestra.run.smithi168.stdout: bdev debug aio = true 2020-11-29T15:55:07.376 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug misdirected ops = true 2020-11-29T15:55:07.376 INFO:teuthology.orchestra.run.smithi168.stdout: debug filestore = 20 2020-11-29T15:55:07.377 INFO:teuthology.orchestra.run.smithi168.stdout: debug journal = 20 2020-11-29T15:55:07.377 INFO:teuthology.orchestra.run.smithi168.stdout: debug ms = 20 2020-11-29T15:55:07.377 INFO:teuthology.orchestra.run.smithi168.stdout: debug osd = 25 2020-11-29T15:55:07.377 INFO:teuthology.orchestra.run.smithi168.stdout: osd fast shutdown = False 2020-11-29T15:55:07.377 INFO:teuthology.orchestra.run.smithi168.stdout: rocksdb delete range threshold = 0 2020-11-29T15:55:07.378 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.378 INFO:teuthology.orchestra.run.smithi168.stdout:[mgr] 2020-11-29T15:55:07.378 INFO:teuthology.orchestra.run.smithi168.stdout: debug ms = 1 2020-11-29T15:55:07.378 INFO:teuthology.orchestra.run.smithi168.stdout: debug mgr = 20 2020-11-29T15:55:07.378 INFO:teuthology.orchestra.run.smithi168.stdout: debug mon = 20 2020-11-29T15:55:07.379 INFO:teuthology.orchestra.run.smithi168.stdout: debug auth = 20 2020-11-29T15:55:07.379 INFO:teuthology.orchestra.run.smithi168.stdout: mon reweight min pgs per osd = 4 2020-11-29T15:55:07.379 INFO:teuthology.orchestra.run.smithi168.stdout: mon reweight min bytes per osd = 10 2020-11-29T15:55:07.379 INFO:teuthology.orchestra.run.smithi168.stdout: mgr/telemetry/nag = false 2020-11-29T15:55:07.379 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.380 INFO:teuthology.orchestra.run.smithi168.stdout:[mon] 2020-11-29T15:55:07.380 INFO:teuthology.orchestra.run.smithi168.stdout: debug ms = 1 2020-11-29T15:55:07.380 INFO:teuthology.orchestra.run.smithi168.stdout: debug mon = 20 2020-11-29T15:55:07.380 INFO:teuthology.orchestra.run.smithi168.stdout: debug paxos = 20 2020-11-29T15:55:07.380 INFO:teuthology.orchestra.run.smithi168.stdout: debug auth = 20 2020-11-29T15:55:07.381 INFO:teuthology.orchestra.run.smithi168.stdout: mon data avail warn = 5 2020-11-29T15:55:07.381 INFO:teuthology.orchestra.run.smithi168.stdout: mon mgr mkfs grace = 240 2020-11-29T15:55:07.381 INFO:teuthology.orchestra.run.smithi168.stdout: mon reweight min pgs per osd = 4 2020-11-29T15:55:07.381 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd reporter subtree level = osd 2020-11-29T15:55:07.381 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd prime pg temp = true 2020-11-29T15:55:07.382 INFO:teuthology.orchestra.run.smithi168.stdout: mon reweight min bytes per osd = 10 2020-11-29T15:55:07.382 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd crush smoke test = False 2020-11-29T15:55:07.382 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:07.382 INFO:teuthology.orchestra.run.smithi168.stdout:[client] 2020-11-29T15:55:07.383 INFO:teuthology.orchestra.run.smithi168.stdout: rgw cache enabled = true 2020-11-29T15:55:07.383 INFO:teuthology.orchestra.run.smithi168.stdout: rgw enable ops log = true 2020-11-29T15:55:07.383 INFO:teuthology.orchestra.run.smithi168.stdout: rgw enable usage log = true 2020-11-29T15:55:07.383 INFO:teuthology.orchestra.run.smithi168.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-29T15:55:07.383 INFO:teuthology.orchestra.run.smithi168.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-29T15:55:07.384 INFO:teuthology.orchestra.run.smithi168.stdout: debug rgw = 20 2020-11-29T15:55:07.384 INFO:teuthology.orchestra.run.smithi168.stdout: rgw crypt require ssl = False 2020-11-29T15:55:07.384 INFO:teuthology.orchestra.run.smithi168.stdout: rgw crypt s3 kms backend = testing 2020-11-29T15:55:07.384 INFO:teuthology.orchestra.run.smithi168.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-29T15:55:07.384 INFO:teuthology.orchestra.run.smithi168.stdout: rgw curl low speed time = 300 2020-11-29T15:55:07.385 INFO:teuthology.orchestra.run.smithi168.stdout: rgw data log num shards = 4 2020-11-29T15:55:07.385 INFO:teuthology.orchestra.run.smithi168.stdout: rgw md log max shards = 4 2020-11-29T15:55:07.385 INFO:teuthology.orchestra.run.smithi168.stdout: rgw sync log trim interval = 0 2020-11-29T15:55:07.385 INFO:teuthology.orchestra.run.smithi168.stdout: rgw sync obj etag verify = True 2020-11-29T15:55:07.385 INFO:teuthology.orchestra.run.smithi168.stdout: setgroup = ceph 2020-11-29T15:55:07.386 INFO:teuthology.orchestra.run.smithi168.stdout: setuser = ceph 2020-11-29T15:55:07.386 INFO:teuthology.orchestra.run.smithi168.stdout:[mon.a] 2020-11-29T15:55:07.386 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.0] 2020-11-29T15:55:07.386 INFO:teuthology.orchestra.run.smithi168.stdout: osd_max_omap_entries_per_request = 10 2020-11-29T15:55:07.386 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.1] 2020-11-29T15:55:07.387 INFO:teuthology.orchestra.run.smithi168.stdout: osd_max_omap_entries_per_request = 1000 2020-11-29T15:55:07.387 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.2] 2020-11-29T15:55:07.387 INFO:teuthology.orchestra.run.smithi168.stdout: osd_max_omap_entries_per_request = 10000 2020-11-29T15:55:07.389 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2020-11-29T15:55:07.389 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2020-11-29T15:55:07.389 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-29T15:55:07.389 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2020-11-29T15:55:07.390 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.390 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2020-11-29T15:55:07.390 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.390 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2020-11-29T15:55:07.391 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.391 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2020-11-29T15:55:07.391 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2020-11-29T15:55:07.391 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.391 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2020-11-29T15:55:07.392 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2020-11-29T15:55:07.392 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.392 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-29T15:55:07.392 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2020-11-29T15:55:07.392 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.393 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2020-11-29T15:55:07.393 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-29T15:55:07.393 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.393 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2020-11-29T15:55:07.393 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.394 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2020-11-29T15:55:07.394 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2020-11-29T15:55:07.394 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2020-11-29T15:55:07.394 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2020-11-29T15:55:07.395 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2020-11-29T15:55:07.395 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2020-11-29T15:55:07.395 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2020-11-29T15:55:07.395 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-29T15:55:07.395 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-29T15:55:07.395 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2020-11-29T15:55:07.396 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.396 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-29T15:55:07.396 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.396 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2020-11-29T15:55:07.396 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.397 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2020-11-29T15:55:07.397 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.397 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2020-11-29T15:55:07.397 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2020-11-29T15:55:07.397 INFO:teuthology.orchestra.run.smithi184.stdout: mon health detail to clog = false 2020-11-29T15:55:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = 172.21.15.168 2020-11-29T15:55:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: osd heartbeat grace = 40 2020-11-29T15:55:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = 73679a84-1a68-4bdd-a7a1-215c2cc5bd23 2020-11-29T15:55:07.398 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.398 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2020-11-29T15:55:07.399 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2020-11-29T15:55:07.399 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.399 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2020-11-29T15:55:07.399 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2020-11-29T15:55:07.400 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.400 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2020-11-29T15:55:07.400 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2020-11-29T15:55:07.400 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.400 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2020-11-29T15:55:07.401 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2020-11-29T15:55:07.401 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2020-11-29T15:55:07.401 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.401 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2020-11-29T15:55:07.401 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2020-11-29T15:55:07.402 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.402 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2020-11-29T15:55:07.402 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.402 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2020-11-29T15:55:07.403 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.403 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2020-11-29T15:55:07.403 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.403 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2020-11-29T15:55:07.403 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2020-11-29T15:55:07.403 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.404 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2020-11-29T15:55:07.404 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2020-11-29T15:55:07.404 INFO:teuthology.orchestra.run.smithi184.stdout: debug filestore = 20 2020-11-29T15:55:07.404 INFO:teuthology.orchestra.run.smithi184.stdout: debug journal = 20 2020-11-29T15:55:07.405 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 20 2020-11-29T15:55:07.405 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 25 2020-11-29T15:55:07.405 INFO:teuthology.orchestra.run.smithi184.stdout: osd fast shutdown = False 2020-11-29T15:55:07.405 INFO:teuthology.orchestra.run.smithi184.stdout: rocksdb delete range threshold = 0 2020-11-29T15:55:07.405 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.405 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2020-11-29T15:55:07.406 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2020-11-29T15:55:07.406 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2020-11-29T15:55:07.406 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2020-11-29T15:55:07.406 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2020-11-29T15:55:07.407 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2020-11-29T15:55:07.407 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2020-11-29T15:55:07.407 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2020-11-29T15:55:07.407 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.407 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2020-11-29T15:55:07.408 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2020-11-29T15:55:07.408 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2020-11-29T15:55:07.408 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2020-11-29T15:55:07.408 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2020-11-29T15:55:07.408 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2020-11-29T15:55:07.409 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2020-11-29T15:55:07.409 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2020-11-29T15:55:07.409 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2020-11-29T15:55:07.409 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2020-11-29T15:55:07.409 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2020-11-29T15:55:07.410 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd crush smoke test = False 2020-11-29T15:55:07.410 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:07.410 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2020-11-29T15:55:07.410 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2020-11-29T15:55:07.410 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2020-11-29T15:55:07.411 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2020-11-29T15:55:07.411 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-29T15:55:07.411 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-29T15:55:07.411 INFO:teuthology.orchestra.run.smithi184.stdout: debug rgw = 20 2020-11-29T15:55:07.412 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt require ssl = False 2020-11-29T15:55:07.412 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms backend = testing 2020-11-29T15:55:07.412 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-29T15:55:07.412 INFO:teuthology.orchestra.run.smithi184.stdout: rgw curl low speed time = 300 2020-11-29T15:55:07.412 INFO:teuthology.orchestra.run.smithi184.stdout: rgw data log num shards = 4 2020-11-29T15:55:07.413 INFO:teuthology.orchestra.run.smithi184.stdout: rgw md log max shards = 4 2020-11-29T15:55:07.413 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync log trim interval = 0 2020-11-29T15:55:07.413 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync obj etag verify = True 2020-11-29T15:55:07.413 INFO:teuthology.orchestra.run.smithi184.stdout: setgroup = ceph 2020-11-29T15:55:07.413 INFO:teuthology.orchestra.run.smithi184.stdout: setuser = ceph 2020-11-29T15:55:07.414 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2020-11-29T15:55:07.414 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2020-11-29T15:55:07.414 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10 2020-11-29T15:55:07.414 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2020-11-29T15:55:07.415 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 1000 2020-11-29T15:55:07.415 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2020-11-29T15:55:07.415 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10000 2020-11-29T15:55:07.415 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-11-29T15:55:07.416 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-11-29T15:55:07.478 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-29T15:55:07.479 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:07.479 INFO:teuthology.orchestra.run.smithi168:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-29T15:55:07.525 INFO:teuthology.orchestra.run.smithi168.stderr:+ dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-11-29T15:55:07.526 INFO:teuthology.orchestra.run.smithi168.stdout:[mon.] 2020-11-29T15:55:07.527 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBbxMNfWoKAAxAAbX6nxNzH5R8Kv5JeTwhhUQ== 2020-11-29T15:55:07.527 INFO:teuthology.orchestra.run.smithi168.stdout:[client.admin] 2020-11-29T15:55:07.527 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBbxMNfYmemHBAAAa7k3mAUivxeULbx2H5Pxw== 2020-11-29T15:55:07.527 INFO:teuthology.orchestra.run.smithi168.stdout: caps mds = "allow *" 2020-11-29T15:55:07.528 INFO:teuthology.orchestra.run.smithi168.stdout: caps mgr = "allow *" 2020-11-29T15:55:07.528 INFO:teuthology.orchestra.run.smithi168.stdout: caps mon = "allow *" 2020-11-29T15:55:07.528 INFO:teuthology.orchestra.run.smithi168.stdout: caps osd = "allow *" 2020-11-29T15:55:07.528 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:07.528 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:07.529 INFO:teuthology.orchestra.run.smithi168.stderr:206 bytes copied, 0.000167431 s, 1.2 MB/s 2020-11-29T15:55:07.529 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:07.529 INFO:teuthology.orchestra.run.smithi168:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-29T15:55:07.573 INFO:teuthology.orchestra.run.smithi168.stderr:+ dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-11-29T15:55:07.575 INFO:teuthology.orchestra.run.smithi168.stdout: �sg��hKݧ�!\,Ž#[��_F�;[��_F�;�aXa �����a 2020-11-29T15:55:07.575 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:07.575 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:07.576 INFO:teuthology.orchestra.run.smithi168.stderr:201 bytes copied, 0.000156774 s, 1.3 MB/s 2020-11-29T15:55:07.576 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi168.front.sepia.ceph.com 2020-11-29T15:55:07.576 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:07.577 INFO:teuthology.orchestra.run.smithi168:> sudo dd of=/etc/ceph/c1.keyring 2020-11-29T15:55:07.577 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-29T15:55:07.621 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-29T15:55:07.630 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:07.631 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:07.631 INFO:teuthology.orchestra.run.smithi168.stderr:206 bytes copied, 0.000127561 s, 1.6 MB/s 2020-11-29T15:55:07.632 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-29T15:55:07.641 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:07.642 INFO:teuthology.orchestra.run.smithi168:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:07.689 INFO:teuthology.orchestra.run.smithi168.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:07.690 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:07.691 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:07.691 INFO:teuthology.orchestra.run.smithi168.stderr:201 bytes copied, 0.000139478 s, 1.4 MB/s 2020-11-29T15:55:07.691 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2020-11-29T15:55:07.692 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:07.692 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/c1.keyring 2020-11-29T15:55:07.692 INFO:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-29T15:55:07.697 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-11-29T15:55:07.706 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:07.707 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:07.707 INFO:teuthology.orchestra.run.smithi184.stderr:206 bytes copied, 6.6766e-05 s, 3.1 MB/s 2020-11-29T15:55:07.707 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-11-29T15:55:07.716 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:07.716 INFO:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:07.764 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:07.766 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:07.767 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:07.767 INFO:teuthology.orchestra.run.smithi184.stderr:201 bytes copied, 7.639e-05 s, 2.6 MB/s 2020-11-29T15:55:07.767 INFO:tasks.ceph:Setting up mon nodes... 2020-11-29T15:55:07.768 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-29T15:55:07.769 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-11-29T15:55:07.813 INFO:teuthology.orchestra.run.smithi168.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-11-29T15:55:07.815 INFO:tasks.ceph:Setting up mds nodes... 2020-11-29T15:55:07.816 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-29T15:55:07.817 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-11-29T15:55:07.889 INFO:teuthology.orchestra.run.smithi168.stdout:creating /etc/ceph/c1.client.0.keyring 2020-11-29T15:55:07.900 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-11-29T15:55:07.973 INFO:teuthology.orchestra.run.smithi168.stdout:creating /etc/ceph/c1.client.1.keyring 2020-11-29T15:55:07.985 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-29T15:55:07.986 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi168.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-29T15:55:07.987 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-11-29T15:55:08.051 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-29T15:55:08.052 INFO:tasks.ceph:role: c1.osd.0 2020-11-29T15:55:08.052 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi168.front.sepia.ceph.com 2020-11-29T15:55:08.053 INFO:teuthology.orchestra.run.smithi168:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-29T15:55:08.587 INFO:teuthology.orchestra.run.smithi168.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-29T15:55:08.588 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-29T15:55:08.588 INFO:teuthology.orchestra.run.smithi168.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-29T15:55:08.589 INFO:teuthology.orchestra.run.smithi168.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-29T15:55:08.589 INFO:teuthology.orchestra.run.smithi168.stdout: = sunit=0 swidth=0 blks 2020-11-29T15:55:08.590 INFO:teuthology.orchestra.run.smithi168.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-29T15:55:08.590 INFO:teuthology.orchestra.run.smithi168.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-29T15:55:08.590 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-29T15:55:08.590 INFO:teuthology.orchestra.run.smithi168.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-29T15:55:08.591 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi168.front.sepia.ceph.com -o noatime 2020-11-29T15:55:08.592 INFO:teuthology.orchestra.run.smithi168:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-11-29T15:55:08.668 INFO:teuthology.orchestra.run.smithi168:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-11-29T15:55:08.679 INFO:teuthology.orchestra.run.smithi168.stderr:sudo: /sbin/restorecon: command not found 2020-11-29T15:55:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-29T15:55:08.680 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-11-29T15:55:08.739 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-29T15:55:08.739 INFO:tasks.ceph:role: c1.osd.1 2020-11-29T15:55:08.739 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi168.front.sepia.ceph.com 2020-11-29T15:55:08.740 INFO:teuthology.orchestra.run.smithi168:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-29T15:55:09.260 INFO:teuthology.orchestra.run.smithi168.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-29T15:55:09.260 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-29T15:55:09.261 INFO:teuthology.orchestra.run.smithi168.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-29T15:55:09.261 INFO:teuthology.orchestra.run.smithi168.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-29T15:55:09.261 INFO:teuthology.orchestra.run.smithi168.stdout: = sunit=0 swidth=0 blks 2020-11-29T15:55:09.262 INFO:teuthology.orchestra.run.smithi168.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-29T15:55:09.262 INFO:teuthology.orchestra.run.smithi168.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-29T15:55:09.262 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-29T15:55:09.263 INFO:teuthology.orchestra.run.smithi168.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-29T15:55:09.264 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi168.front.sepia.ceph.com -o noatime 2020-11-29T15:55:09.264 INFO:teuthology.orchestra.run.smithi168:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-11-29T15:55:09.328 INFO:teuthology.orchestra.run.smithi168:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-11-29T15:55:09.338 INFO:teuthology.orchestra.run.smithi168.stderr:sudo: /sbin/restorecon: command not found 2020-11-29T15:55:09.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-29T15:55:09.339 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-11-29T15:55:09.395 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-29T15:55:09.395 INFO:tasks.ceph:role: c1.osd.2 2020-11-29T15:55:09.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi168.front.sepia.ceph.com 2020-11-29T15:55:09.396 INFO:teuthology.orchestra.run.smithi168:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-29T15:55:09.895 INFO:teuthology.orchestra.run.smithi168.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-29T15:55:09.895 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-29T15:55:09.895 INFO:teuthology.orchestra.run.smithi168.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-29T15:55:09.896 INFO:teuthology.orchestra.run.smithi168.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-29T15:55:09.896 INFO:teuthology.orchestra.run.smithi168.stdout: = sunit=0 swidth=0 blks 2020-11-29T15:55:09.896 INFO:teuthology.orchestra.run.smithi168.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-29T15:55:09.896 INFO:teuthology.orchestra.run.smithi168.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-29T15:55:09.897 INFO:teuthology.orchestra.run.smithi168.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-29T15:55:09.897 INFO:teuthology.orchestra.run.smithi168.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-29T15:55:09.898 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi168.front.sepia.ceph.com -o noatime 2020-11-29T15:55:09.898 INFO:teuthology.orchestra.run.smithi168:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-11-29T15:55:09.938 INFO:teuthology.orchestra.run.smithi168:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-11-29T15:55:09.992 INFO:teuthology.orchestra.run.smithi168.stderr:sudo: /sbin/restorecon: command not found 2020-11-29T15:55:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-29T15:55:09.993 INFO:teuthology.orchestra.run.smithi168:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:10.081 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:10.084+0000 7fd3c3a14d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-11-29T15:55:10.082 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:10.084+0000 7fd3c3a14d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-11-29T15:55:10.082 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:10.084+0000 7fd3c3a14d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-29T15:55:10.083 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:10.084+0000 7fd3c3a14d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-29T15:55:10.083 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:10.084+0000 7fd3c3a14d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-11-29T15:55:10.088 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:10.088+0000 7fd3c3a14d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-11-29T15:55:12.249 INFO:teuthology.orchestra.run.smithi168:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-11-29T15:55:12.265 INFO:teuthology.orchestra.run.smithi168:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:12.347 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:12.348+0000 7f1f42368d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-11-29T15:55:12.348 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:12.348+0000 7f1f42368d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-11-29T15:55:12.348 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:12.348+0000 7f1f42368d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-29T15:55:12.348 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:12.348+0000 7f1f42368d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-29T15:55:12.348 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:12.348+0000 7f1f42368d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-11-29T15:55:12.349 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:12.348+0000 7f1f42368d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-11-29T15:55:14.548 INFO:teuthology.orchestra.run.smithi168:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-11-29T15:55:14.608 INFO:teuthology.orchestra.run.smithi168:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:14.693 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:14.696+0000 7f843883dd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-11-29T15:55:14.693 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:14.696+0000 7f843883dd80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-11-29T15:55:14.694 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:14.696+0000 7f843883dd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-29T15:55:14.694 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:14.696+0000 7f843883dd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-29T15:55:14.694 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:14.696+0000 7f843883dd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-11-29T15:55:14.695 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:55:14.696+0000 7f843883dd80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-11-29T15:55:16.860 INFO:teuthology.orchestra.run.smithi168:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-11-29T15:55:16.922 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-29T15:55:16.922 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:16.923 INFO:teuthology.orchestra.run.smithi168:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-29T15:55:16.969 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-11-29T15:55:16.977 INFO:teuthology.orchestra.run.smithi168.stdout:[mgr.x] 2020-11-29T15:55:16.978 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBbxMNfld25MBAAoo952bYqZtfT8y3HMVSGuA== 2020-11-29T15:55:16.978 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:16.978 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:16.979 INFO:teuthology.orchestra.run.smithi168.stderr:56 bytes copied, 0.00016815 s, 333 kB/s 2020-11-29T15:55:16.979 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:16.980 INFO:teuthology.orchestra.run.smithi168:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-29T15:55:17.025 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-11-29T15:55:17.033 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.0] 2020-11-29T15:55:17.034 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBexMNfSg0eBRAAzLRI4OQSonyGR7Ziu8WJSg== 2020-11-29T15:55:17.034 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:17.034 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:17.035 INFO:teuthology.orchestra.run.smithi168.stderr:56 bytes copied, 0.000160342 s, 349 kB/s 2020-11-29T15:55:17.036 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:17.036 INFO:teuthology.orchestra.run.smithi168:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-29T15:55:17.081 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-11-29T15:55:17.089 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.1] 2020-11-29T15:55:17.090 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBgxMNfLrvvFBAAKpIBfJvPsMlQfy5fk6h1Mg== 2020-11-29T15:55:17.090 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:17.091 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:17.091 INFO:teuthology.orchestra.run.smithi168.stderr:56 bytes copied, 0.000164177 s, 341 kB/s 2020-11-29T15:55:17.092 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:17.092 INFO:teuthology.orchestra.run.smithi168:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-29T15:55:17.141 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-11-29T15:55:17.149 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.2] 2020-11-29T15:55:17.149 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBixMNf+niWKRAAFVumGZdiSfS0j+f8/BtjeA== 2020-11-29T15:55:17.149 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:17.150 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:17.150 INFO:teuthology.orchestra.run.smithi168.stderr:56 bytes copied, 0.000165554 s, 338 kB/s 2020-11-29T15:55:17.151 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:17.151 INFO:teuthology.orchestra.run.smithi168:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-29T15:55:17.197 INFO:teuthology.orchestra.run.smithi168.stderr:+ dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-11-29T15:55:17.198 INFO:teuthology.orchestra.run.smithi168.stdout:[client.0] 2020-11-29T15:55:17.198 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBbxMNfAuhCNRAAlbxR5S6bIyOx/ZUETwJa2A== 2020-11-29T15:55:17.198 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:17.198 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:17.199 INFO:teuthology.orchestra.run.smithi168.stderr:59 bytes copied, 0.000161004 s, 366 kB/s 2020-11-29T15:55:17.199 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:17.199 INFO:teuthology.orchestra.run.smithi168:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-29T15:55:17.245 INFO:teuthology.orchestra.run.smithi168.stderr:+ dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-11-29T15:55:17.246 INFO:teuthology.orchestra.run.smithi168.stdout:[client.1] 2020-11-29T15:55:17.246 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBbxMNf4FtGOhAAh1h85QccaYOEmPxsY+bhoA== 2020-11-29T15:55:17.246 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:17.246 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:17.247 INFO:teuthology.orchestra.run.smithi168.stderr:59 bytes copied, 0.000162338 s, 363 kB/s 2020-11-29T15:55:17.247 INFO:tasks.ceph:Adding keys to all mons... 2020-11-29T15:55:17.247 INFO:teuthology.orchestra.run.smithi168:> sudo tee -a /etc/ceph/c1.keyring 2020-11-29T15:55:17.301 INFO:teuthology.orchestra.run.smithi168.stdout:[mgr.x] 2020-11-29T15:55:17.301 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBbxMNfld25MBAAoo952bYqZtfT8y3HMVSGuA== 2020-11-29T15:55:17.302 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.0] 2020-11-29T15:55:17.302 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBexMNfSg0eBRAAzLRI4OQSonyGR7Ziu8WJSg== 2020-11-29T15:55:17.302 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.1] 2020-11-29T15:55:17.302 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBgxMNfLrvvFBAAKpIBfJvPsMlQfy5fk6h1Mg== 2020-11-29T15:55:17.302 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.2] 2020-11-29T15:55:17.303 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBixMNf+niWKRAAFVumGZdiSfS0j+f8/BtjeA== 2020-11-29T15:55:17.303 INFO:teuthology.orchestra.run.smithi168.stdout:[client.0] 2020-11-29T15:55:17.303 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBbxMNfAuhCNRAAlbxR5S6bIyOx/ZUETwJa2A== 2020-11-29T15:55:17.303 INFO:teuthology.orchestra.run.smithi168.stdout:[client.1] 2020-11-29T15:55:17.303 INFO:teuthology.orchestra.run.smithi168.stdout: key = AQBbxMNf4FtGOhAAh1h85QccaYOEmPxsY+bhoA== 2020-11-29T15:55:17.304 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-29T15:55:17.393 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-29T15:55:17.478 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-29T15:55:17.562 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-29T15:55:17.646 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-29T15:55:17.733 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-29T15:55:17.821 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-29T15:55:17.821 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-11-29T15:55:17.880 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-11-29T15:55:17.975 INFO:teuthology.orchestra.run.smithi168:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-11-29T15:55:17.987 INFO:teuthology.orchestra.run.smithi168:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-11-29T15:55:18.034 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-11-29T15:55:18.035 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-11-29T15:55:18.035 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-11-29T15:55:18.079 INFO:tasks.ceph.c1.mon.a:Started 2020-11-29T15:55:18.079 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-11-29T15:55:18.080 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-11-29T15:55:18.081 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-11-29T15:55:18.084 INFO:tasks.ceph.c1.mgr.x:Started 2020-11-29T15:55:18.085 INFO:tasks.ceph:Setting crush tunables to default 2020-11-29T15:55:18.085 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c1 osd crush tunables default 2020-11-29T15:55:18.555 INFO:teuthology.orchestra.run.smithi168.stderr:adjusted tunables profile to default 2020-11-29T15:55:18.565 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-11-29T15:55:18.566 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:18.566 INFO:teuthology.orchestra.run.smithi168:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-29T15:55:18.569 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-11-29T15:55:18.574 INFO:teuthology.orchestra.run.smithi168.stdout:3ee28a05-772f-4981-80dc-76ef280699a5 2020-11-29T15:55:18.574 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:18.574 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:18.575 INFO:teuthology.orchestra.run.smithi168.stderr:37 bytes copied, 0.000157192 s, 235 kB/s 2020-11-29T15:55:18.576 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:18.576 INFO:teuthology.orchestra.run.smithi168:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-29T15:55:18.620 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-11-29T15:55:18.627 INFO:teuthology.orchestra.run.smithi168.stdout:b843f07d-6cfe-4c32-9997-5fb2d60a58ff 2020-11-29T15:55:18.628 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:18.628 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:18.628 INFO:teuthology.orchestra.run.smithi168.stderr:37 bytes copied, 0.000130893 s, 283 kB/s 2020-11-29T15:55:18.629 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:18.629 INFO:teuthology.orchestra.run.smithi168:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-29T15:55:18.677 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-11-29T15:55:18.685 INFO:teuthology.orchestra.run.smithi168.stdout:2bf8f64c-0c3f-4294-87f7-d304b1805ea8 2020-11-29T15:55:18.685 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:18.686 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:18.686 INFO:teuthology.orchestra.run.smithi168.stderr:37 bytes copied, 0.000180375 s, 205 kB/s 2020-11-29T15:55:18.687 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c1 osd new 3ee28a05-772f-4981-80dc-76ef280699a5 0 2020-11-29T15:55:18.941 INFO:teuthology.orchestra.run.smithi168.stdout:0 2020-11-29T15:55:18.954 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c1 osd new b843f07d-6cfe-4c32-9997-5fb2d60a58ff 1 2020-11-29T15:55:19.159 INFO:teuthology.orchestra.run.smithi168.stdout:1 2020-11-29T15:55:19.171 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c1 osd new 2bf8f64c-0c3f-4294-87f7-d304b1805ea8 2 2020-11-29T15:55:19.386 INFO:teuthology.orchestra.run.smithi168.stdout:2 2020-11-29T15:55:19.399 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-11-29T15:55:19.399 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-11-29T15:55:19.401 INFO:tasks.ceph.c1.osd.0:Started 2020-11-29T15:55:19.402 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-11-29T15:55:19.402 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-11-29T15:55:19.404 INFO:tasks.ceph.c1.osd.1:Started 2020-11-29T15:55:19.405 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-11-29T15:55:19.405 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-11-29T15:55:19.407 INFO:tasks.ceph.c1.osd.2:Started 2020-11-29T15:55:19.407 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-29T15:55:19.711 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-29T15:55:19.897 INFO:teuthology.misc.health.smithi168.stdout: 2020-11-29T15:55:19.897 INFO:teuthology.misc.health.smithi168.stdout:{"epoch":5,"fsid":"73679a84-1a68-4bdd-a7a1-215c2cc5bd23","created":"2020-11-29T15:55:18.199473+0000","modified":"2020-11-29T15:55:19.388482+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"3ee28a05-772f-4981-80dc-76ef280699a5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b843f07d-6cfe-4c32-9997-5fb2d60a58ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2bf8f64c-0c3f-4294-87f7-d304b1805ea8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:19.913 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-29T15:55:20.014 INFO:tasks.ceph.c1.osd.0.smithi168.stderr:2020-11-29T15:55:20.016+0000 7f988181bd80 -1 Falling back to public interface 2020-11-29T15:55:20.014 INFO:tasks.ceph.c1.osd.1.smithi168.stderr:2020-11-29T15:55:20.016+0000 7f691741cd80 -1 Falling back to public interface 2020-11-29T15:55:20.015 INFO:tasks.ceph.c1.osd.2.smithi168.stderr:2020-11-29T15:55:20.016+0000 7ff46916ad80 -1 Falling back to public interface 2020-11-29T15:55:20.645 INFO:tasks.ceph.c1.osd.2.smithi168.stderr:2020-11-29T15:55:20.648+0000 7ff46916ad80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-29T15:55:20.650 INFO:tasks.ceph.c1.osd.1.smithi168.stderr:2020-11-29T15:55:20.652+0000 7f691741cd80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-29T15:55:20.691 INFO:tasks.ceph.c1.osd.0.smithi168.stderr:2020-11-29T15:55:20.692+0000 7f988181bd80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-29T15:55:22.229 INFO:tasks.ceph.c1.osd.2.smithi168.stderr:2020-11-29T15:55:22.228+0000 7ff461e37700 -1 osd.2 0 waiting for initial osdmap 2020-11-29T15:55:22.229 INFO:tasks.ceph.c1.osd.0.smithi168.stderr:2020-11-29T15:55:22.232+0000 7f9878ce5700 -1 osd.0 0 waiting for initial osdmap 2020-11-29T15:55:22.230 INFO:tasks.ceph.c1.osd.1.smithi168.stderr:2020-11-29T15:55:22.232+0000 7f690e8e6700 -1 osd.1 0 waiting for initial osdmap 2020-11-29T15:55:26.217 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-11-29T15:55:26.503 INFO:teuthology.misc.health.smithi168.stdout: 2020-11-29T15:55:26.503 INFO:teuthology.misc.health.smithi168.stdout:{"epoch":10,"fsid":"73679a84-1a68-4bdd-a7a1-215c2cc5bd23","created":"2020-11-29T15:55:18.199473+0000","modified":"2020-11-29T15:55:25.253565+0000","last_up_change":"2020-11-29T15:55:23.225453+0000","last_in_change":"2020-11-29T15:55:23.225453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-29T15:55:22.135044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"3ee28a05-772f-4981-80dc-76ef280699a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6800","nonce":12517},{"type":"v1","addr":"172.21.15.168:6801","nonce":12517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6802","nonce":12517},{"type":"v1","addr":"172.21.15.168:6805","nonce":12517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6814","nonce":12517},{"type":"v1","addr":"172.21.15.168:6816","nonce":12517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6808","nonce":12517},{"type":"v1","addr":"172.21.15.168:6811","nonce":12517}]},"public_addr":"172.21.15.168:6801/12517","cluster_addr":"172.21.15.168:6805/12517","heartbeat_back_addr":"172.21.15.168:6816/12517","heartbeat_front_addr":"172.21.15.168:6811/12517","state":["exists","up"]},{"osd":1,"uuid":"b843f07d-6cfe-4c32-9997-5fb2d60a58ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6803","nonce":12518},{"type":"v1","addr":"172.21.15.168:6804","nonce":12518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6806","nonce":12518},{"type":"v1","addr":"172.21.15.168:6807","nonce":12518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6817","nonce":12518},{"type":"v1","addr":"172.21.15.168:6819","nonce":12518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6809","nonce":12518},{"type":"v1","addr":"172.21.15.168:6812","nonce":12518}]},"public_addr":"172.21.15.168:6804/12518","cluster_addr":"172.21.15.168:6807/12518","heartbeat_back_addr":"172.21.15.168:6819/12518","heartbeat_front_addr":"172.21.15.168:6812/12518","state":["exists","up"]},{"osd":2,"uuid":"2bf8f64c-0c3f-4294-87f7-d304b1805ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6810","nonce":12519},{"type":"v1","addr":"172.21.15.168:6813","nonce":12519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6815","nonce":12519},{"type":"v1","addr":"172.21.15.168:6818","nonce":12519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6822","nonce":12519},{"type":"v1","addr":"172.21.15.168:6823","nonce":12519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6820","nonce":12519},{"type":"v1","addr":"172.21.15.168:6821","nonce":12519}]},"public_addr":"172.21.15.168:6813/12519","cluster_addr":"172.21.15.168:6818/12519","heartbeat_back_addr":"172.21.15.168:6823/12519","heartbeat_front_addr":"172.21.15.168:6821/12519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.686207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.618275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.628051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:26.516 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-29T15:55:26.516 INFO:tasks.ceph:Creating RBD pool 2020-11-29T15:55:26.517 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c1 osd pool create rbd 8 2020-11-29T15:55:27.290 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'rbd' created 2020-11-29T15:55:27.304 INFO:teuthology.orchestra.run.smithi168:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-29T15:55:28.298 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-29T15:55:28.313 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-11-29T15:55:28.314 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-29T15:55:28.314 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-29T15:55:28.591 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:28.591 INFO:teuthology.orchestra.run.smithi168.stdout:{"epoch":12,"fsid":"73679a84-1a68-4bdd-a7a1-215c2cc5bd23","created":"2020-11-29T15:55:18.199473+0000","modified":"2020-11-29T15:55:28.278484+0000","last_up_change":"2020-11-29T15:55:23.225453+0000","last_in_change":"2020-11-29T15:55:23.225453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-29T15:55:22.135044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-29T15:55:26.804636+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3ee28a05-772f-4981-80dc-76ef280699a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6800","nonce":12517},{"type":"v1","addr":"172.21.15.168:6801","nonce":12517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6802","nonce":12517},{"type":"v1","addr":"172.21.15.168:6805","nonce":12517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6814","nonce":12517},{"type":"v1","addr":"172.21.15.168:6816","nonce":12517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6808","nonce":12517},{"type":"v1","addr":"172.21.15.168:6811","nonce":12517}]},"public_addr":"172.21.15.168:6801/12517","cluster_addr":"172.21.15.168:6805/12517","heartbeat_back_addr":"172.21.15.168:6816/12517","heartbeat_front_addr":"172.21.15.168:6811/12517","state":["exists","up"]},{"osd":1,"uuid":"b843f07d-6cfe-4c32-9997-5fb2d60a58ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6803","nonce":12518},{"type":"v1","addr":"172.21.15.168:6804","nonce":12518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6806","nonce":12518},{"type":"v1","addr":"172.21.15.168:6807","nonce":12518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6817","nonce":12518},{"type":"v1","addr":"172.21.15.168:6819","nonce":12518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6809","nonce":12518},{"type":"v1","addr":"172.21.15.168:6812","nonce":12518}]},"public_addr":"172.21.15.168:6804/12518","cluster_addr":"172.21.15.168:6807/12518","heartbeat_back_addr":"172.21.15.168:6819/12518","heartbeat_front_addr":"172.21.15.168:6812/12518","state":["exists","up"]},{"osd":2,"uuid":"2bf8f64c-0c3f-4294-87f7-d304b1805ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6810","nonce":12519},{"type":"v1","addr":"172.21.15.168:6813","nonce":12519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6815","nonce":12519},{"type":"v1","addr":"172.21.15.168:6818","nonce":12519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6822","nonce":12519},{"type":"v1","addr":"172.21.15.168:6823","nonce":12519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6820","nonce":12519},{"type":"v1","addr":"172.21.15.168:6821","nonce":12519}]},"public_addr":"172.21.15.168:6813/12519","cluster_addr":"172.21.15.168:6818/12519","heartbeat_back_addr":"172.21.15.168:6823/12519","heartbeat_front_addr":"172.21.15.168:6821/12519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.686207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.618275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.628051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:28.605 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-29T15:55:22.135044+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-29T15:55:26.804636+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-29T15:55:28.606 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2020-11-29T15:55:28.897 INFO:teuthology.orchestra.run.smithi168.stdout:pg_num: 1 2020-11-29T15:55:28.911 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2020-11-29T15:55:29.205 INFO:teuthology.orchestra.run.smithi168.stdout:pg_num: 8 2020-11-29T15:55:29.224 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-11-29T15:55:29.224 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-11-29T15:55:29.224 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2020-11-29T15:55:29.553 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:29.567 INFO:teuthology.orchestra.run.smithi168.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6824","nonce":12334},{"type":"v1","addr":"172.21.15.168:6825","nonce":12334}]},"active_addr":"172.21.15.168:6825/12334","active_change":"2020-11-29T15:55:22.099463+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.168:0","nonce":1118808845}]},{"addrvec":[{"type":"v2","addr":"172.21.15.168:0","nonce":1763460094}]},{"addrvec":[{"type":"v2","addr":"172.21.15.168:0","nonce":42756245}]}]}} 2020-11-29T15:55:29.570 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-11-29T15:55:29.570 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-11-29T15:55:29.570 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-29T15:55:29.866 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:29.866 INFO:teuthology.orchestra.run.smithi168.stdout:{"epoch":13,"fsid":"73679a84-1a68-4bdd-a7a1-215c2cc5bd23","created":"2020-11-29T15:55:18.199473+0000","modified":"2020-11-29T15:55:29.288061+0000","last_up_change":"2020-11-29T15:55:23.225453+0000","last_in_change":"2020-11-29T15:55:23.225453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-29T15:55:22.135044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-29T15:55:26.804636+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3ee28a05-772f-4981-80dc-76ef280699a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6800","nonce":12517},{"type":"v1","addr":"172.21.15.168:6801","nonce":12517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6802","nonce":12517},{"type":"v1","addr":"172.21.15.168:6805","nonce":12517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6814","nonce":12517},{"type":"v1","addr":"172.21.15.168:6816","nonce":12517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6808","nonce":12517},{"type":"v1","addr":"172.21.15.168:6811","nonce":12517}]},"public_addr":"172.21.15.168:6801/12517","cluster_addr":"172.21.15.168:6805/12517","heartbeat_back_addr":"172.21.15.168:6816/12517","heartbeat_front_addr":"172.21.15.168:6811/12517","state":["exists","up"]},{"osd":1,"uuid":"b843f07d-6cfe-4c32-9997-5fb2d60a58ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6803","nonce":12518},{"type":"v1","addr":"172.21.15.168:6804","nonce":12518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6806","nonce":12518},{"type":"v1","addr":"172.21.15.168:6807","nonce":12518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6817","nonce":12518},{"type":"v1","addr":"172.21.15.168:6819","nonce":12518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6809","nonce":12518},{"type":"v1","addr":"172.21.15.168:6812","nonce":12518}]},"public_addr":"172.21.15.168:6804/12518","cluster_addr":"172.21.15.168:6807/12518","heartbeat_back_addr":"172.21.15.168:6819/12518","heartbeat_front_addr":"172.21.15.168:6812/12518","state":["exists","up"]},{"osd":2,"uuid":"2bf8f64c-0c3f-4294-87f7-d304b1805ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6810","nonce":12519},{"type":"v1","addr":"172.21.15.168:6813","nonce":12519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6815","nonce":12519},{"type":"v1","addr":"172.21.15.168:6818","nonce":12519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6822","nonce":12519},{"type":"v1","addr":"172.21.15.168:6823","nonce":12519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6820","nonce":12519},{"type":"v1","addr":"172.21.15.168:6821","nonce":12519}]},"public_addr":"172.21.15.168:6813/12519","cluster_addr":"172.21.15.168:6818/12519","heartbeat_back_addr":"172.21.15.168:6823/12519","heartbeat_front_addr":"172.21.15.168:6821/12519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.686207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.618275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.628051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:29.886 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-11-29T15:55:29.886 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2020-11-29T15:55:30.172 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:30.173 INFO:teuthology.orchestra.run.smithi168.stdout:{"epoch":13,"fsid":"73679a84-1a68-4bdd-a7a1-215c2cc5bd23","created":"2020-11-29T15:55:18.199473+0000","modified":"2020-11-29T15:55:29.288061+0000","last_up_change":"2020-11-29T15:55:23.225453+0000","last_in_change":"2020-11-29T15:55:23.225453+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-29T15:55:22.135044+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-29T15:55:26.804636+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3ee28a05-772f-4981-80dc-76ef280699a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6800","nonce":12517},{"type":"v1","addr":"172.21.15.168:6801","nonce":12517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6802","nonce":12517},{"type":"v1","addr":"172.21.15.168:6805","nonce":12517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6814","nonce":12517},{"type":"v1","addr":"172.21.15.168:6816","nonce":12517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6808","nonce":12517},{"type":"v1","addr":"172.21.15.168:6811","nonce":12517}]},"public_addr":"172.21.15.168:6801/12517","cluster_addr":"172.21.15.168:6805/12517","heartbeat_back_addr":"172.21.15.168:6816/12517","heartbeat_front_addr":"172.21.15.168:6811/12517","state":["exists","up"]},{"osd":1,"uuid":"b843f07d-6cfe-4c32-9997-5fb2d60a58ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6803","nonce":12518},{"type":"v1","addr":"172.21.15.168:6804","nonce":12518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6806","nonce":12518},{"type":"v1","addr":"172.21.15.168:6807","nonce":12518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6817","nonce":12518},{"type":"v1","addr":"172.21.15.168:6819","nonce":12518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6809","nonce":12518},{"type":"v1","addr":"172.21.15.168:6812","nonce":12518}]},"public_addr":"172.21.15.168:6804/12518","cluster_addr":"172.21.15.168:6807/12518","heartbeat_back_addr":"172.21.15.168:6819/12518","heartbeat_front_addr":"172.21.15.168:6812/12518","state":["exists","up"]},{"osd":2,"uuid":"2bf8f64c-0c3f-4294-87f7-d304b1805ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6810","nonce":12519},{"type":"v1","addr":"172.21.15.168:6813","nonce":12519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6815","nonce":12519},{"type":"v1","addr":"172.21.15.168:6818","nonce":12519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6822","nonce":12519},{"type":"v1","addr":"172.21.15.168:6823","nonce":12519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.168:6820","nonce":12519},{"type":"v1","addr":"172.21.15.168:6821","nonce":12519}]},"public_addr":"172.21.15.168:6813/12519","cluster_addr":"172.21.15.168:6818/12519","heartbeat_back_addr":"172.21.15.168:6823/12519","heartbeat_front_addr":"172.21.15.168:6821/12519","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.686207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.618275+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:21.628051+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:30.192 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2020-11-29T15:55:30.354 INFO:teuthology.orchestra.run.smithi168.stdout:34359738372 2020-11-29T15:55:30.354 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2020-11-29T15:55:30.522 INFO:teuthology.orchestra.run.smithi168.stdout:34359738372 2020-11-29T15:55:30.523 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2020-11-29T15:55:30.690 INFO:teuthology.orchestra.run.smithi168.stdout:34359738372 2020-11-29T15:55:30.691 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-29T15:55:30.973 INFO:teuthology.orchestra.run.smithi168.stdout:34359738371 2020-11-29T15:55:30.988 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2020-11-29T15:55:31.989 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-29T15:55:32.288 INFO:teuthology.orchestra.run.smithi168.stdout:34359738371 2020-11-29T15:55:32.303 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2020-11-29T15:55:33.305 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2020-11-29T15:55:33.597 INFO:teuthology.orchestra.run.smithi168.stdout:34359738372 2020-11-29T15:55:33.612 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2020-11-29T15:55:33.612 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2020-11-29T15:55:33.899 INFO:teuthology.orchestra.run.smithi168.stdout:34359738372 2020-11-29T15:55:33.914 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2020-11-29T15:55:33.914 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2020-11-29T15:55:34.213 INFO:teuthology.orchestra.run.smithi168.stdout:34359738372 2020-11-29T15:55:34.227 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2020-11-29T15:55:34.228 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-11-29T15:55:34.228 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-29T15:55:34.511 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:34.514 INFO:teuthology.orchestra.run.smithi168.stderr:dumped all 2020-11-29T15:55:34.525 INFO:teuthology.orchestra.run.smithi168.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-29T15:55:34.109123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14744,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558056,"statfs":{"total":322122547200,"available":322107449344,"internally_reserved":0,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.815347"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.306591+0000","last_change":"2020-11-29T15:55:28.305262+0000","last_active":"2020-11-29T15:55:28.306591+0000","last_peered":"2020-11-29T15:55:28.306591+0000","last_clean":"2020-11-29T15:55:28.306591+0000","last_became_active":"2020-11-29T15:55:28.304791+0000","last_became_peered":"2020-11-29T15:55:28.304791+0000","last_unstale":"2020-11-29T15:55:28.306591+0000","last_undegraded":"2020-11-29T15:55:28.306591+0000","last_fullsized":"2020-11-29T15:55:28.306591+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.306016+0000","last_change":"2020-11-29T15:55:28.303936+0000","last_active":"2020-11-29T15:55:28.306016+0000","last_peered":"2020-11-29T15:55:28.306016+0000","last_clean":"2020-11-29T15:55:28.306016+0000","last_became_active":"2020-11-29T15:55:28.303469+0000","last_became_peered":"2020-11-29T15:55:28.303469+0000","last_unstale":"2020-11-29T15:55:28.306016+0000","last_undegraded":"2020-11-29T15:55:28.306016+0000","last_fullsized":"2020-11-29T15:55:28.306016+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.306009+0000","last_change":"2020-11-29T15:55:28.304809+0000","last_active":"2020-11-29T15:55:28.306009+0000","last_peered":"2020-11-29T15:55:28.306009+0000","last_clean":"2020-11-29T15:55:28.306009+0000","last_became_active":"2020-11-29T15:55:28.304369+0000","last_became_peered":"2020-11-29T15:55:28.304369+0000","last_unstale":"2020-11-29T15:55:28.306009+0000","last_undegraded":"2020-11-29T15:55:28.306009+0000","last_fullsized":"2020-11-29T15:55:28.306009+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.303926+0000","last_change":"2020-11-29T15:55:28.302851+0000","last_active":"2020-11-29T15:55:28.303926+0000","last_peered":"2020-11-29T15:55:28.303926+0000","last_clean":"2020-11-29T15:55:28.303926+0000","last_became_active":"2020-11-29T15:55:28.302343+0000","last_became_peered":"2020-11-29T15:55:28.302343+0000","last_unstale":"2020-11-29T15:55:28.303926+0000","last_undegraded":"2020-11-29T15:55:28.303926+0000","last_fullsized":"2020-11-29T15:55:28.303926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.296672+0000","last_change":"2020-11-29T15:55:24.758598+0000","last_active":"2020-11-29T15:55:28.296672+0000","last_peered":"2020-11-29T15:55:28.296672+0000","last_clean":"2020-11-29T15:55:28.296672+0000","last_became_active":"2020-11-29T15:55:24.756963+0000","last_became_peered":"2020-11-29T15:55:24.756963+0000","last_unstale":"2020-11-29T15:55:28.296672+0000","last_undegraded":"2020-11-29T15:55:28.296672+0000","last_fullsized":"2020-11-29T15:55:28.296672+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:22.217780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:22.217780+0000","last_clean_scrub_stamp":"2020-11-29T15:55:22.217780+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.303835+0000","last_change":"2020-11-29T15:55:28.301758+0000","last_active":"2020-11-29T15:55:28.303835+0000","last_peered":"2020-11-29T15:55:28.303835+0000","last_clean":"2020-11-29T15:55:28.303835+0000","last_became_active":"2020-11-29T15:55:28.301220+0000","last_became_peered":"2020-11-29T15:55:28.301220+0000","last_unstale":"2020-11-29T15:55:28.303835+0000","last_undegraded":"2020-11-29T15:55:28.303835+0000","last_fullsized":"2020-11-29T15:55:28.303835+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.301478+0000","last_change":"2020-11-29T15:55:28.299790+0000","last_active":"2020-11-29T15:55:28.301478+0000","last_peered":"2020-11-29T15:55:28.301478+0000","last_clean":"2020-11-29T15:55:28.301478+0000","last_became_active":"2020-11-29T15:55:28.299173+0000","last_became_peered":"2020-11-29T15:55:28.299173+0000","last_unstale":"2020-11-29T15:55:28.301478+0000","last_undegraded":"2020-11-29T15:55:28.301478+0000","last_fullsized":"2020-11-29T15:55:28.301478+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.308510+0000","last_change":"2020-11-29T15:55:28.307319+0000","last_active":"2020-11-29T15:55:28.308510+0000","last_peered":"2020-11-29T15:55:28.308510+0000","last_clean":"2020-11-29T15:55:28.308510+0000","last_became_active":"2020-11-29T15:55:28.306930+0000","last_became_peered":"2020-11-29T15:55:28.306930+0000","last_unstale":"2020-11-29T15:55:28.308510+0000","last_undegraded":"2020-11-29T15:55:28.308510+0000","last_fullsized":"2020-11-29T15:55:28.308510+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.305824+0000","last_change":"2020-11-29T15:55:28.303727+0000","last_active":"2020-11-29T15:55:28.305824+0000","last_peered":"2020-11-29T15:55:28.305824+0000","last_clean":"2020-11-29T15:55:28.305824+0000","last_became_active":"2020-11-29T15:55:28.303241+0000","last_became_peered":"2020-11-29T15:55:28.303241+0000","last_unstale":"2020-11-29T15:55:28.305824+0000","last_undegraded":"2020-11-29T15:55:28.305824+0000","last_fullsized":"2020-11-29T15:55:28.305824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]}]}} 2020-11-29T15:55:34.527 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2020-11-29T15:55:34.817 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:34.819 INFO:teuthology.orchestra.run.smithi168.stderr:dumped all 2020-11-29T15:55:34.831 INFO:teuthology.orchestra.run.smithi168.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-11-29T15:55:34.109123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14744,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558056,"statfs":{"total":322122547200,"available":322107449344,"internally_reserved":0,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.815347"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.306591+0000","last_change":"2020-11-29T15:55:28.305262+0000","last_active":"2020-11-29T15:55:28.306591+0000","last_peered":"2020-11-29T15:55:28.306591+0000","last_clean":"2020-11-29T15:55:28.306591+0000","last_became_active":"2020-11-29T15:55:28.304791+0000","last_became_peered":"2020-11-29T15:55:28.304791+0000","last_unstale":"2020-11-29T15:55:28.306591+0000","last_undegraded":"2020-11-29T15:55:28.306591+0000","last_fullsized":"2020-11-29T15:55:28.306591+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.306016+0000","last_change":"2020-11-29T15:55:28.303936+0000","last_active":"2020-11-29T15:55:28.306016+0000","last_peered":"2020-11-29T15:55:28.306016+0000","last_clean":"2020-11-29T15:55:28.306016+0000","last_became_active":"2020-11-29T15:55:28.303469+0000","last_became_peered":"2020-11-29T15:55:28.303469+0000","last_unstale":"2020-11-29T15:55:28.306016+0000","last_undegraded":"2020-11-29T15:55:28.306016+0000","last_fullsized":"2020-11-29T15:55:28.306016+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.306009+0000","last_change":"2020-11-29T15:55:28.304809+0000","last_active":"2020-11-29T15:55:28.306009+0000","last_peered":"2020-11-29T15:55:28.306009+0000","last_clean":"2020-11-29T15:55:28.306009+0000","last_became_active":"2020-11-29T15:55:28.304369+0000","last_became_peered":"2020-11-29T15:55:28.304369+0000","last_unstale":"2020-11-29T15:55:28.306009+0000","last_undegraded":"2020-11-29T15:55:28.306009+0000","last_fullsized":"2020-11-29T15:55:28.306009+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.303926+0000","last_change":"2020-11-29T15:55:28.302851+0000","last_active":"2020-11-29T15:55:28.303926+0000","last_peered":"2020-11-29T15:55:28.303926+0000","last_clean":"2020-11-29T15:55:28.303926+0000","last_became_active":"2020-11-29T15:55:28.302343+0000","last_became_peered":"2020-11-29T15:55:28.302343+0000","last_unstale":"2020-11-29T15:55:28.303926+0000","last_undegraded":"2020-11-29T15:55:28.303926+0000","last_fullsized":"2020-11-29T15:55:28.303926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.296672+0000","last_change":"2020-11-29T15:55:24.758598+0000","last_active":"2020-11-29T15:55:28.296672+0000","last_peered":"2020-11-29T15:55:28.296672+0000","last_clean":"2020-11-29T15:55:28.296672+0000","last_became_active":"2020-11-29T15:55:24.756963+0000","last_became_peered":"2020-11-29T15:55:24.756963+0000","last_unstale":"2020-11-29T15:55:28.296672+0000","last_undegraded":"2020-11-29T15:55:28.296672+0000","last_fullsized":"2020-11-29T15:55:28.296672+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:22.217780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:22.217780+0000","last_clean_scrub_stamp":"2020-11-29T15:55:22.217780+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.303835+0000","last_change":"2020-11-29T15:55:28.301758+0000","last_active":"2020-11-29T15:55:28.303835+0000","last_peered":"2020-11-29T15:55:28.303835+0000","last_clean":"2020-11-29T15:55:28.303835+0000","last_became_active":"2020-11-29T15:55:28.301220+0000","last_became_peered":"2020-11-29T15:55:28.301220+0000","last_unstale":"2020-11-29T15:55:28.303835+0000","last_undegraded":"2020-11-29T15:55:28.303835+0000","last_fullsized":"2020-11-29T15:55:28.303835+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.301478+0000","last_change":"2020-11-29T15:55:28.299790+0000","last_active":"2020-11-29T15:55:28.301478+0000","last_peered":"2020-11-29T15:55:28.301478+0000","last_clean":"2020-11-29T15:55:28.301478+0000","last_became_active":"2020-11-29T15:55:28.299173+0000","last_became_peered":"2020-11-29T15:55:28.299173+0000","last_unstale":"2020-11-29T15:55:28.301478+0000","last_undegraded":"2020-11-29T15:55:28.301478+0000","last_fullsized":"2020-11-29T15:55:28.301478+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.308510+0000","last_change":"2020-11-29T15:55:28.307319+0000","last_active":"2020-11-29T15:55:28.308510+0000","last_peered":"2020-11-29T15:55:28.308510+0000","last_clean":"2020-11-29T15:55:28.308510+0000","last_became_active":"2020-11-29T15:55:28.306930+0000","last_became_peered":"2020-11-29T15:55:28.306930+0000","last_unstale":"2020-11-29T15:55:28.308510+0000","last_undegraded":"2020-11-29T15:55:28.308510+0000","last_fullsized":"2020-11-29T15:55:28.308510+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:28.305824+0000","last_change":"2020-11-29T15:55:28.303727+0000","last_active":"2020-11-29T15:55:28.305824+0000","last_peered":"2020-11-29T15:55:28.305824+0000","last_clean":"2020-11-29T15:55:28.305824+0000","last_became_active":"2020-11-29T15:55:28.303241+0000","last_became_peered":"2020-11-29T15:55:28.303241+0000","last_unstale":"2020-11-29T15:55:28.305824+0000","last_undegraded":"2020-11-29T15:55:28.305824+0000","last_fullsized":"2020-11-29T15:55:28.305824+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:27.270089+0000","last_clean_scrub_stamp":"2020-11-29T15:55:27.270089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.407}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]}]}} 2020-11-29T15:55:34.832 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-11-29T15:55:34.832 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-11-29T15:55:34.833 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-11-29T15:55:34.833 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2020-11-29T15:55:35.170 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:35.183 INFO:teuthology.orchestra.run.smithi168.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-29T15:55:35.184 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-11-29T15:55:35.184 INFO:teuthology.run_tasks:Running task ceph... 2020-11-29T15:55:35.196 INFO:tasks.ceph:Creating ceph cluster c2... 2020-11-29T15:55:35.196 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-29T15:55:35.197 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi/5665427', 'branch': 'wip-rgw-d3n-cache-upstreaming_i20', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'mkogan@redhat.com', 'first_in_suite': False, 'job_id': '5665427', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mkogan-2020-11-29_15:41:09-rgw-wip-rgw-d3n-cache-upstreaming_i20-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-rgw-d3n-cache-upstreaming_i20'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-rgw-d3n-cache-upstreaming_i20', 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6'}}, 'owner': 'scheduled_mkogan@teuthology', 'priority': 150, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-d3n-cache-upstreaming_i20', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_wip-rgw-d3n-cache-upstreaming_i20/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '10684a45d9b8cdfc272d8721720e890aa44052c6', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-29_15:41:09', 'tube': 'smithi', 'user': 'mkogan', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.25223', 'targets': {'smithi168.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdrYZbb8MPSUJwpZ8Q7VVCJhkVlWIs7odbYMnfk3AU5J446G411sRshkndZuW7PE3fXPGJD8BInXKRBhBSy+nkRiv9n/om6poKcIG5Yv1SdpNdOtxTpOXzBjpOeYYLVHrUpq/cza4Nh+IBhyhavgDiU5L0pP215lQHswHSTZPAkTMzNuWQQ2QhjV+d9kzDSfggdzVyU9Xj4q3SE+MQU6Aptm3J/Qhrxj11C8hUB2LYJGFFqKw8lWW50ylVaCByHuuLLmQhHRhl5Ysgpe5Rnnpk7GF10RN3yvQxmeW6XEl1ZrKGdk232VZWx7BGHW3+VF8DiLW6t0umenxaKPyTTX8V', 'smithi184.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDadENPl2I5HwceuElxWd2qNVQqbzVzle1/SaPOx61ShU45ELWegCqyHgUkO4iCRopi05bspLcoF0hyQ7DncvrWAcmlw1mimGxjM+6UgakidoKNBba9EZ5FTG2O9nEXoNFYIe5DXoA+/QcrShkDQnTKWuKsb41jXhS1UrPNCK+wEZ98w5ClbEL8YMMMFdH6LNGyizsXvmFWKK4VLPhWlqGVa2zi+P8P28XoUn3fI8OvQ8DTnlDhRZAbuduRfopcjIKERfn2SU2z+0dmj8bUoTL5vS809ClPLFR2xTprUkI/dmRJE4JGUb+E/ns79IIn+AG2FCys0qZ0YGaLrmTzPPXT'}} 2020-11-29T15:55:35.198 INFO:teuthology.orchestra.run.smithi168:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-29T15:55:35.200 INFO:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-11-29T15:55:35.208 INFO:teuthology.orchestra.run.smithi168:> sudo install -d -m0777 -- /var/run/ceph 2020-11-29T15:55:35.246 INFO:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2020-11-29T15:55:35.266 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:35.266 INFO:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2020-11-29T15:55:35.311 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-29T15:55:35.312 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_1 2020-11-29T15:55:35.313 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_2 2020-11-29T15:55:35.313 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_3 2020-11-29T15:55:35.313 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_4 2020-11-29T15:55:35.313 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:35.314 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:35.314 INFO:teuthology.orchestra.run.smithi184.stderr:72 bytes copied, 0.00017472 s, 412 kB/s 2020-11-29T15:55:35.314 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-29T15:55:35.315 INFO:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-29T15:55:35.361 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-29T15:55:35.362 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-29T15:55:35.362 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 6h/6d Inode: 340 Links: 1 2020-11-29T15:55:35.362 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-29T15:55:35.362 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2020-11-29 15:52:46.235434463 +0000 2020-11-29T15:55:35.363 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2020-11-29 15:52:46.151392466 +0000 2020-11-29T15:55:35.363 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2020-11-29 15:52:46.151392466 +0000 2020-11-29T15:55:35.363 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2020-11-29T15:55:35.371 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2020-11-29T15:55:35.371 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2020-11-29T15:55:35.372 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000380494 s, 1.3 MB/s 2020-11-29T15:55:35.375 INFO:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-29T15:55:35.425 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 6h/6d Inode: 347 Links: 1 2020-11-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-29T15:55:35.426 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2020-11-29 15:52:46.471552459 +0000 2020-11-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2020-11-29 15:52:46.383508460 +0000 2020-11-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2020-11-29 15:52:46.383508460 +0000 2020-11-29T15:55:35.427 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2020-11-29T15:55:35.434 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2020-11-29T15:55:35.434 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2020-11-29T15:55:35.434 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000304084 s, 1.7 MB/s 2020-11-29T15:55:35.438 INFO:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-29T15:55:35.485 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-29T15:55:35.486 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-29T15:55:35.486 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 6h/6d Inode: 353 Links: 1 2020-11-29T15:55:35.486 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-29T15:55:35.486 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2020-11-29 15:52:46.723678453 +0000 2020-11-29T15:55:35.487 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2020-11-29 15:52:46.639636454 +0000 2020-11-29T15:55:35.487 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2020-11-29 15:52:46.639636454 +0000 2020-11-29T15:55:35.487 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2020-11-29T15:55:35.495 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2020-11-29T15:55:35.495 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2020-11-29T15:55:35.495 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00037171 s, 1.4 MB/s 2020-11-29T15:55:35.499 INFO:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-29T15:55:35.549 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-29T15:55:35.550 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-29T15:55:35.550 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 6h/6d Inode: 359 Links: 1 2020-11-29T15:55:35.550 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-29T15:55:35.550 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2020-11-29 15:52:46.867750450 +0000 2020-11-29T15:55:35.551 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2020-11-29 15:52:46.863748450 +0000 2020-11-29T15:55:35.551 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2020-11-29 15:52:46.863748450 +0000 2020-11-29T15:55:35.551 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2020-11-29T15:55:35.559 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2020-11-29T15:55:35.559 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2020-11-29T15:55:35.559 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000413059 s, 1.2 MB/s 2020-11-29T15:55:35.563 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-29T15:55:35.563 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-29T15:55:35.564 INFO:tasks.ceph:Generating config... 2020-11-29T15:55:35.565 INFO:tasks.ceph:[client] debug rgw = 20 2020-11-29T15:55:35.565 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-11-29T15:55:35.565 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-11-29T15:55:35.566 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-29T15:55:35.566 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-11-29T15:55:35.566 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-11-29T15:55:35.567 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-11-29T15:55:35.567 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-11-29T15:55:35.567 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-11-29T15:55:35.567 INFO:tasks.ceph:[client] setgroup = ceph 2020-11-29T15:55:35.568 INFO:tasks.ceph:[client] setuser = ceph 2020-11-29T15:55:35.568 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-11-29T15:55:35.568 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-29T15:55:35.569 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-29T15:55:35.569 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-29T15:55:35.569 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-29T15:55:35.569 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-29T15:55:35.570 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-11-29T15:55:35.570 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-29T15:55:35.570 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-29T15:55:35.571 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-29T15:55:35.571 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-29T15:55:35.571 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-11-29T15:55:35.571 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-11-29T15:55:35.572 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-11-29T15:55:35.572 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-11-29T15:55:35.572 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-11-29T15:55:35.573 INFO:tasks.ceph:Setting up c2.mon.a... 2020-11-29T15:55:35.573 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-11-29T15:55:35.639 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.keyring 2020-11-29T15:55:35.641 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-11-29T15:55:35.724 INFO:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-29T15:55:35.781 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.184')] 2020-11-29T15:55:35.782 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.184', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-11-29T15:55:35.783 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:35.783 INFO:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-29T15:55:35.827 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-29T15:55:35.828 INFO:teuthology.orchestra.run.smithi184.stderr:6+1 records in 2020-11-29T15:55:35.829 INFO:teuthology.orchestra.run.smithi184.stderr:6+1 records out 2020-11-29T15:55:35.829 INFO:teuthology.orchestra.run.smithi184.stderr:3136 bytes (3.1 kB, 3.1 KiB) copied, 0.000107331 s, 29.2 MB/s 2020-11-29T15:55:35.829 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.184 --print /home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:35.900 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:55:35.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:55:35.902 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:35.903 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: generated fsid ebcec895-1108-492d-9d57-61e3df7f12c1 2020-11-29T15:55:35.903 INFO:teuthology.orchestra.run.smithi184.stdout:epoch 0 2020-11-29T15:55:35.903 INFO:teuthology.orchestra.run.smithi184.stdout:fsid ebcec895-1108-492d-9d57-61e3df7f12c1 2020-11-29T15:55:35.904 INFO:teuthology.orchestra.run.smithi184.stdout:last_changed 2020-11-29T15:55:35.906135+0000 2020-11-29T15:55:35.904 INFO:teuthology.orchestra.run.smithi184.stdout:created 2020-11-29T15:55:35.906135+0000 2020-11-29T15:55:35.905 INFO:teuthology.orchestra.run.smithi184.stdout:min_mon_release 0 (unknown) 2020-11-29T15:55:35.905 INFO:teuthology.orchestra.run.smithi184.stdout:election_strategy: 1 2020-11-29T15:55:35.905 INFO:teuthology.orchestra.run.smithi184.stdout:0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2020-11-29T15:55:35.906 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-11-29T15:55:35.907 INFO:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-29T15:55:35.956 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID ebcec895-1108-492d-9d57-61e3df7f12c1... 2020-11-29T15:55:35.958 INFO:teuthology.orchestra.run.smithi168:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-29T15:55:35.961 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-11-29T15:55:36.001 INFO:teuthology.orchestra.run.smithi168.stdout:[global] 2020-11-29T15:55:36.002 INFO:teuthology.orchestra.run.smithi168.stdout: chdir = "" 2020-11-29T15:55:36.002 INFO:teuthology.orchestra.run.smithi168.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-29T15:55:36.002 INFO:teuthology.orchestra.run.smithi168.stdout: auth supported = cephx 2020-11-29T15:55:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: filestore xattr use omap = true 2020-11-29T15:55:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: mon clock drift allowed = 1.000 2020-11-29T15:55:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: osd crush chooseleaf type = 0 2020-11-29T15:55:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: auth debug = true 2020-11-29T15:55:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: ms die on old message = true 2020-11-29T15:55:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: ms die on bug = true 2020-11-29T15:55:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-29T15:55:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: mon pg warn max object skew = 0 2020-11-29T15:55:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: # disable pg_autoscaler by default for new pools 2020-11-29T15:55:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-29T15:55:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: osd pool default size = 2 2020-11-29T15:55:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.007 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd allow primary affinity = true 2020-11-29T15:55:36.007 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd allow pg remap = true 2020-11-29T15:55:36.007 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on legacy crush tunables = false 2020-11-29T15:55:36.007 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on crush straw calc version zero = false 2020-11-29T15:55:36.007 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on no sortbitwise = false 2020-11-29T15:55:36.008 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on osd down out interval zero = false 2020-11-29T15:55:36.008 INFO:teuthology.orchestra.run.smithi168.stdout: mon warn on too few osds = false 2020-11-29T15:55:36.008 INFO:teuthology.orchestra.run.smithi168.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-29T15:55:36.008 INFO:teuthology.orchestra.run.smithi168.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-29T15:55:36.008 INFO:teuthology.orchestra.run.smithi168.stdout: mon_allow_pool_size_one = true 2020-11-29T15:55:36.009 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.009 INFO:teuthology.orchestra.run.smithi168.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-29T15:55:36.009 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.009 INFO:teuthology.orchestra.run.smithi168.stdout: osd default data pool replay window = 5 2020-11-29T15:55:36.009 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.010 INFO:teuthology.orchestra.run.smithi168.stdout: mon allow pool delete = true 2020-11-29T15:55:36.010 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.010 INFO:teuthology.orchestra.run.smithi168.stdout: mon cluster log file level = debug 2020-11-29T15:55:36.011 INFO:teuthology.orchestra.run.smithi168.stdout: debug asserts on shutdown = true 2020-11-29T15:55:36.011 INFO:teuthology.orchestra.run.smithi168.stdout: mon health detail to clog = false 2020-11-29T15:55:36.011 INFO:teuthology.orchestra.run.smithi168.stdout: mon host = 172.21.15.184 2020-11-29T15:55:36.011 INFO:teuthology.orchestra.run.smithi168.stdout: osd heartbeat grace = 40 2020-11-29T15:55:36.011 INFO:teuthology.orchestra.run.smithi168.stdout: fsid = ebcec895-1108-492d-9d57-61e3df7f12c1 2020-11-29T15:55:36.012 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.012 INFO:teuthology.orchestra.run.smithi168.stdout:[osd] 2020-11-29T15:55:36.012 INFO:teuthology.orchestra.run.smithi168.stdout: osd journal size = 100 2020-11-29T15:55:36.012 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.013 INFO:teuthology.orchestra.run.smithi168.stdout: osd scrub load threshold = 5.0 2020-11-29T15:55:36.013 INFO:teuthology.orchestra.run.smithi168.stdout: osd scrub max interval = 600 2020-11-29T15:55:36.013 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.013 INFO:teuthology.orchestra.run.smithi168.stdout: osd recover clone overlap = true 2020-11-29T15:55:36.013 INFO:teuthology.orchestra.run.smithi168.stdout: osd recovery max chunk = 1048576 2020-11-29T15:55:36.014 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.014 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug shutdown = true 2020-11-29T15:55:36.014 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug op order = true 2020-11-29T15:55:36.014 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug verify stray on activate = true 2020-11-29T15:55:36.014 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.015 INFO:teuthology.orchestra.run.smithi168.stdout: osd open classes on start = true 2020-11-29T15:55:36.015 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug pg log writeout = true 2020-11-29T15:55:36.015 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.015 INFO:teuthology.orchestra.run.smithi168.stdout: osd deep scrub update digest min age = 30 2020-11-29T15:55:36.015 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.016 INFO:teuthology.orchestra.run.smithi168.stdout: osd map max advance = 10 2020-11-29T15:55:36.016 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.016 INFO:teuthology.orchestra.run.smithi168.stdout: journal zero on create = true 2020-11-29T15:55:36.016 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.017 INFO:teuthology.orchestra.run.smithi168.stdout: filestore ondisk finisher threads = 3 2020-11-29T15:55:36.017 INFO:teuthology.orchestra.run.smithi168.stdout: filestore apply finisher threads = 3 2020-11-29T15:55:36.017 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.017 INFO:teuthology.orchestra.run.smithi168.stdout: bdev debug aio = true 2020-11-29T15:55:36.017 INFO:teuthology.orchestra.run.smithi168.stdout: osd debug misdirected ops = true 2020-11-29T15:55:36.018 INFO:teuthology.orchestra.run.smithi168.stdout: debug filestore = 20 2020-11-29T15:55:36.018 INFO:teuthology.orchestra.run.smithi168.stdout: debug journal = 20 2020-11-29T15:55:36.018 INFO:teuthology.orchestra.run.smithi168.stdout: debug ms = 20 2020-11-29T15:55:36.018 INFO:teuthology.orchestra.run.smithi168.stdout: debug osd = 25 2020-11-29T15:55:36.019 INFO:teuthology.orchestra.run.smithi168.stdout: osd fast shutdown = False 2020-11-29T15:55:36.019 INFO:teuthology.orchestra.run.smithi168.stdout: rocksdb delete range threshold = 0 2020-11-29T15:55:36.019 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.019 INFO:teuthology.orchestra.run.smithi168.stdout:[mgr] 2020-11-29T15:55:36.020 INFO:teuthology.orchestra.run.smithi168.stdout: debug ms = 1 2020-11-29T15:55:36.020 INFO:teuthology.orchestra.run.smithi168.stdout: debug mgr = 20 2020-11-29T15:55:36.020 INFO:teuthology.orchestra.run.smithi168.stdout: debug mon = 20 2020-11-29T15:55:36.020 INFO:teuthology.orchestra.run.smithi168.stdout: debug auth = 20 2020-11-29T15:55:36.020 INFO:teuthology.orchestra.run.smithi168.stdout: mon reweight min pgs per osd = 4 2020-11-29T15:55:36.021 INFO:teuthology.orchestra.run.smithi168.stdout: mon reweight min bytes per osd = 10 2020-11-29T15:55:36.021 INFO:teuthology.orchestra.run.smithi168.stdout: mgr/telemetry/nag = false 2020-11-29T15:55:36.021 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.021 INFO:teuthology.orchestra.run.smithi168.stdout:[mon] 2020-11-29T15:55:36.022 INFO:teuthology.orchestra.run.smithi168.stdout: debug ms = 1 2020-11-29T15:55:36.022 INFO:teuthology.orchestra.run.smithi168.stdout: debug mon = 20 2020-11-29T15:55:36.022 INFO:teuthology.orchestra.run.smithi168.stdout: debug paxos = 20 2020-11-29T15:55:36.022 INFO:teuthology.orchestra.run.smithi168.stdout: debug auth = 20 2020-11-29T15:55:36.022 INFO:teuthology.orchestra.run.smithi168.stdout: mon data avail warn = 5 2020-11-29T15:55:36.023 INFO:teuthology.orchestra.run.smithi168.stdout: mon mgr mkfs grace = 240 2020-11-29T15:55:36.023 INFO:teuthology.orchestra.run.smithi168.stdout: mon reweight min pgs per osd = 4 2020-11-29T15:55:36.023 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd reporter subtree level = osd 2020-11-29T15:55:36.023 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd prime pg temp = true 2020-11-29T15:55:36.023 INFO:teuthology.orchestra.run.smithi168.stdout: mon reweight min bytes per osd = 10 2020-11-29T15:55:36.024 INFO:teuthology.orchestra.run.smithi168.stdout: mon osd crush smoke test = False 2020-11-29T15:55:36.024 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:55:36.024 INFO:teuthology.orchestra.run.smithi168.stdout:[client] 2020-11-29T15:55:36.024 INFO:teuthology.orchestra.run.smithi168.stdout: rgw cache enabled = true 2020-11-29T15:55:36.024 INFO:teuthology.orchestra.run.smithi168.stdout: rgw enable ops log = true 2020-11-29T15:55:36.025 INFO:teuthology.orchestra.run.smithi168.stdout: rgw enable usage log = true 2020-11-29T15:55:36.025 INFO:teuthology.orchestra.run.smithi168.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-29T15:55:36.025 INFO:teuthology.orchestra.run.smithi168.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-29T15:55:36.025 INFO:teuthology.orchestra.run.smithi168.stdout: debug rgw = 20 2020-11-29T15:55:36.025 INFO:teuthology.orchestra.run.smithi168.stdout: rgw crypt require ssl = False 2020-11-29T15:55:36.026 INFO:teuthology.orchestra.run.smithi168.stdout: rgw crypt s3 kms backend = testing 2020-11-29T15:55:36.026 INFO:teuthology.orchestra.run.smithi168.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-29T15:55:36.026 INFO:teuthology.orchestra.run.smithi168.stdout: rgw curl low speed time = 300 2020-11-29T15:55:36.026 INFO:teuthology.orchestra.run.smithi168.stdout: rgw data log num shards = 4 2020-11-29T15:55:36.026 INFO:teuthology.orchestra.run.smithi168.stdout: rgw md log max shards = 4 2020-11-29T15:55:36.027 INFO:teuthology.orchestra.run.smithi168.stdout: rgw sync log trim interval = 0 2020-11-29T15:55:36.027 INFO:teuthology.orchestra.run.smithi168.stdout: rgw sync obj etag verify = True 2020-11-29T15:55:36.027 INFO:teuthology.orchestra.run.smithi168.stdout: setgroup = ceph 2020-11-29T15:55:36.027 INFO:teuthology.orchestra.run.smithi168.stdout: setuser = ceph 2020-11-29T15:55:36.027 INFO:teuthology.orchestra.run.smithi168.stdout:[mon.a] 2020-11-29T15:55:36.027 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.0] 2020-11-29T15:55:36.028 INFO:teuthology.orchestra.run.smithi168.stdout: osd_max_omap_entries_per_request = 10 2020-11-29T15:55:36.028 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.1] 2020-11-29T15:55:36.028 INFO:teuthology.orchestra.run.smithi168.stdout: osd_max_omap_entries_per_request = 1000 2020-11-29T15:55:36.028 INFO:teuthology.orchestra.run.smithi168.stdout:[osd.2] 2020-11-29T15:55:36.028 INFO:teuthology.orchestra.run.smithi168.stdout: osd_max_omap_entries_per_request = 10000 2020-11-29T15:55:36.031 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2020-11-29T15:55:36.032 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2020-11-29T15:55:36.032 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-29T15:55:36.032 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2020-11-29T15:55:36.033 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.033 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2020-11-29T15:55:36.034 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.034 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2020-11-29T15:55:36.034 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.035 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2020-11-29T15:55:36.035 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2020-11-29T15:55:36.035 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.035 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2020-11-29T15:55:36.035 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2020-11-29T15:55:36.036 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.036 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-29T15:55:36.036 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2020-11-29T15:55:36.036 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.036 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2020-11-29T15:55:36.037 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-29T15:55:36.037 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.037 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2020-11-29T15:55:36.037 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.038 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2020-11-29T15:55:36.038 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2020-11-29T15:55:36.038 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2020-11-29T15:55:36.038 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2020-11-29T15:55:36.038 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2020-11-29T15:55:36.039 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2020-11-29T15:55:36.039 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2020-11-29T15:55:36.039 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-29T15:55:36.039 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-29T15:55:36.040 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2020-11-29T15:55:36.040 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.040 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-29T15:55:36.040 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.040 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2020-11-29T15:55:36.041 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.041 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2020-11-29T15:55:36.041 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.041 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2020-11-29T15:55:36.041 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2020-11-29T15:55:36.041 INFO:teuthology.orchestra.run.smithi184.stdout: mon health detail to clog = false 2020-11-29T15:55:36.042 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = 172.21.15.184 2020-11-29T15:55:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: osd heartbeat grace = 40 2020-11-29T15:55:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = ebcec895-1108-492d-9d57-61e3df7f12c1 2020-11-29T15:55:36.043 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.043 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2020-11-29T15:55:36.044 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2020-11-29T15:55:36.044 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.044 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2020-11-29T15:55:36.044 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2020-11-29T15:55:36.045 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.045 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2020-11-29T15:55:36.045 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2020-11-29T15:55:36.045 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.045 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2020-11-29T15:55:36.046 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2020-11-29T15:55:36.046 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2020-11-29T15:55:36.046 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.046 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2020-11-29T15:55:36.046 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2020-11-29T15:55:36.047 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.047 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2020-11-29T15:55:36.047 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.047 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2020-11-29T15:55:36.047 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.048 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2020-11-29T15:55:36.048 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.048 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2020-11-29T15:55:36.048 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2020-11-29T15:55:36.048 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.049 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2020-11-29T15:55:36.049 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2020-11-29T15:55:36.049 INFO:teuthology.orchestra.run.smithi184.stdout: debug filestore = 20 2020-11-29T15:55:36.049 INFO:teuthology.orchestra.run.smithi184.stdout: debug journal = 20 2020-11-29T15:55:36.049 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 20 2020-11-29T15:55:36.050 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 25 2020-11-29T15:55:36.050 INFO:teuthology.orchestra.run.smithi184.stdout: osd fast shutdown = False 2020-11-29T15:55:36.050 INFO:teuthology.orchestra.run.smithi184.stdout: rocksdb delete range threshold = 0 2020-11-29T15:55:36.050 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.050 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2020-11-29T15:55:36.051 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2020-11-29T15:55:36.051 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2020-11-29T15:55:36.051 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2020-11-29T15:55:36.051 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2020-11-29T15:55:36.051 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2020-11-29T15:55:36.052 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2020-11-29T15:55:36.052 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2020-11-29T15:55:36.052 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.052 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2020-11-29T15:55:36.052 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2020-11-29T15:55:36.053 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2020-11-29T15:55:36.053 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2020-11-29T15:55:36.053 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2020-11-29T15:55:36.053 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2020-11-29T15:55:36.053 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2020-11-29T15:55:36.054 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2020-11-29T15:55:36.054 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2020-11-29T15:55:36.054 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2020-11-29T15:55:36.054 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2020-11-29T15:55:36.054 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd crush smoke test = False 2020-11-29T15:55:36.055 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:36.055 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2020-11-29T15:55:36.055 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2020-11-29T15:55:36.055 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2020-11-29T15:55:36.055 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2020-11-29T15:55:36.056 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-29T15:55:36.056 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-29T15:55:36.056 INFO:teuthology.orchestra.run.smithi184.stdout: debug rgw = 20 2020-11-29T15:55:36.056 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt require ssl = False 2020-11-29T15:55:36.056 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms backend = testing 2020-11-29T15:55:36.057 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-11-29T15:55:36.057 INFO:teuthology.orchestra.run.smithi184.stdout: rgw curl low speed time = 300 2020-11-29T15:55:36.057 INFO:teuthology.orchestra.run.smithi184.stdout: rgw data log num shards = 4 2020-11-29T15:55:36.057 INFO:teuthology.orchestra.run.smithi184.stdout: rgw md log max shards = 4 2020-11-29T15:55:36.057 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync log trim interval = 0 2020-11-29T15:55:36.058 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync obj etag verify = True 2020-11-29T15:55:36.058 INFO:teuthology.orchestra.run.smithi184.stdout: setgroup = ceph 2020-11-29T15:55:36.058 INFO:teuthology.orchestra.run.smithi184.stdout: setuser = ceph 2020-11-29T15:55:36.058 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2020-11-29T15:55:36.058 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2020-11-29T15:55:36.059 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10 2020-11-29T15:55:36.059 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2020-11-29T15:55:36.059 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 1000 2020-11-29T15:55:36.059 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2020-11-29T15:55:36.059 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10000 2020-11-29T15:55:36.060 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-11-29T15:55:36.061 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-11-29T15:55:36.148 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-29T15:55:36.149 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:36.149 INFO:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-29T15:55:36.152 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-11-29T15:55:36.153 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.] 2020-11-29T15:55:36.153 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB3xMNfEOhNKxAApw8feWn+2Ar18YJ4zdehnw== 2020-11-29T15:55:36.154 INFO:teuthology.orchestra.run.smithi184.stdout:[client.admin] 2020-11-29T15:55:36.154 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB4xMNflxEJCRAAUdRL5i5Dl6Ndms1ZvEIQuA== 2020-11-29T15:55:36.154 INFO:teuthology.orchestra.run.smithi184.stdout: caps mds = "allow *" 2020-11-29T15:55:36.154 INFO:teuthology.orchestra.run.smithi184.stdout: caps mgr = "allow *" 2020-11-29T15:55:36.154 INFO:teuthology.orchestra.run.smithi184.stdout: caps mon = "allow *" 2020-11-29T15:55:36.155 INFO:teuthology.orchestra.run.smithi184.stdout: caps osd = "allow *" 2020-11-29T15:55:36.155 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:36.155 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:36.155 INFO:teuthology.orchestra.run.smithi184.stderr:206 bytes copied, 0.000101858 s, 2.0 MB/s 2020-11-29T15:55:36.156 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:36.156 INFO:teuthology.orchestra.run.smithi184:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-29T15:55:36.200 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-11-29T15:55:36.201 INFO:teuthology.orchestra.run.smithi184.stdout: ���ȕI-�Wa���w��_-�6w��_-�6�aXa �����a 2020-11-29T15:55:36.202 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:36.202 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:36.202 INFO:teuthology.orchestra.run.smithi184.stderr:201 bytes copied, 0.000161017 s, 1.2 MB/s 2020-11-29T15:55:36.203 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi168.front.sepia.ceph.com 2020-11-29T15:55:36.204 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:36.204 INFO:teuthology.orchestra.run.smithi168:> sudo dd of=/etc/ceph/c2.keyring 2020-11-29T15:55:36.205 INFO:teuthology.orchestra.run.smithi168:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-29T15:55:36.211 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-29T15:55:36.220 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:36.221 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:36.221 INFO:teuthology.orchestra.run.smithi168.stderr:206 bytes copied, 7.1375e-05 s, 2.9 MB/s 2020-11-29T15:55:36.222 INFO:teuthology.orchestra.run.smithi168.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-29T15:55:36.231 INFO:teuthology.orchestra.run.smithi168:> set -ex 2020-11-29T15:55:36.232 INFO:teuthology.orchestra.run.smithi168:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:36.277 INFO:teuthology.orchestra.run.smithi168.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:36.278 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records in 2020-11-29T15:55:36.278 INFO:teuthology.orchestra.run.smithi168.stderr:0+1 records out 2020-11-29T15:55:36.279 INFO:teuthology.orchestra.run.smithi168.stderr:201 bytes copied, 6.6549e-05 s, 3.0 MB/s 2020-11-29T15:55:36.279 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2020-11-29T15:55:36.279 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:36.280 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/c2.keyring 2020-11-29T15:55:36.280 INFO:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-29T15:55:36.285 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-11-29T15:55:36.294 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:36.294 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:36.295 INFO:teuthology.orchestra.run.smithi184.stderr:206 bytes copied, 0.00012016 s, 1.7 MB/s 2020-11-29T15:55:36.295 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-11-29T15:55:36.305 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:36.306 INFO:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:36.351 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:36.353 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:36.353 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:36.354 INFO:teuthology.orchestra.run.smithi184.stderr:201 bytes copied, 0.00012876 s, 1.6 MB/s 2020-11-29T15:55:36.354 INFO:tasks.ceph:Setting up mon nodes... 2020-11-29T15:55:36.355 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-29T15:55:36.355 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-11-29T15:55:36.434 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-11-29T15:55:36.436 INFO:tasks.ceph:Setting up mds nodes... 2020-11-29T15:55:36.437 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-29T15:55:36.437 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-11-29T15:55:36.512 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.client.0.keyring 2020-11-29T15:55:36.524 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-11-29T15:55:36.599 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.client.1.keyring 2020-11-29T15:55:36.611 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-29T15:55:36.611 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi168.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-11-29T15:55:36.611 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-11-29T15:55:36.663 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-29T15:55:36.664 INFO:tasks.ceph:role: c2.osd.0 2020-11-29T15:55:36.664 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com 2020-11-29T15:55:36.664 INFO:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-29T15:55:37.175 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-29T15:55:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-29T15:55:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-29T15:55:37.176 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-29T15:55:37.176 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2020-11-29T15:55:37.177 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-29T15:55:37.177 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-29T15:55:37.177 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-29T15:55:37.177 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-29T15:55:37.178 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2020-11-29T15:55:37.179 INFO:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-11-29T15:55:37.256 INFO:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-11-29T15:55:37.266 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2020-11-29T15:55:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-29T15:55:37.268 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-11-29T15:55:37.325 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-29T15:55:37.325 INFO:tasks.ceph:role: c2.osd.1 2020-11-29T15:55:37.325 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com 2020-11-29T15:55:37.326 INFO:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-29T15:55:37.840 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-29T15:55:37.841 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-29T15:55:37.841 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-29T15:55:37.841 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-29T15:55:37.842 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2020-11-29T15:55:37.842 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-29T15:55:37.843 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-29T15:55:37.843 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-29T15:55:37.844 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-29T15:55:37.845 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2020-11-29T15:55:37.846 INFO:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-11-29T15:55:37.920 INFO:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-11-29T15:55:37.932 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2020-11-29T15:55:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-29T15:55:37.933 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-11-29T15:55:37.988 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-11-29T15:55:37.988 INFO:tasks.ceph:role: c2.osd.2 2020-11-29T15:55:37.988 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com 2020-11-29T15:55:37.989 INFO:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-29T15:55:38.504 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-29T15:55:38.504 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-29T15:55:38.505 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-29T15:55:38.505 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-29T15:55:38.505 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2020-11-29T15:55:38.505 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-29T15:55:38.506 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-29T15:55:38.506 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-29T15:55:38.506 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-29T15:55:38.507 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2020-11-29T15:55:38.507 INFO:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-11-29T15:55:38.555 INFO:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-11-29T15:55:38.566 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2020-11-29T15:55:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-29T15:55:38.567 INFO:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:38.661 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:38.664+0000 7fab1d16dd80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-11-29T15:55:38.662 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:38.664+0000 7fab1d16dd80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-11-29T15:55:38.663 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:38.664+0000 7fab1d16dd80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-29T15:55:38.663 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:38.664+0000 7fab1d16dd80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-29T15:55:38.663 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:38.664+0000 7fab1d16dd80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-11-29T15:55:38.666 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:38.668+0000 7fab1d16dd80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-11-29T15:55:40.849 INFO:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-11-29T15:55:40.865 INFO:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:40.943 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:40.944+0000 7fc711e00d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-11-29T15:55:40.943 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:40.944+0000 7fc711e00d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-11-29T15:55:40.944 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:40.944+0000 7fc711e00d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-29T15:55:40.944 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:40.948+0000 7fc711e00d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-29T15:55:40.944 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:40.948+0000 7fc711e00d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-11-29T15:55:40.945 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:40.948+0000 7fc711e00d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-11-29T15:55:43.119 INFO:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-11-29T15:55:43.175 INFO:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:43.253 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:43.256+0000 7f8a46c39d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-11-29T15:55:43.253 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:43.256+0000 7f8a46c39d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-11-29T15:55:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:43.256+0000 7f8a46c39d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-29T15:55:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:43.256+0000 7f8a46c39d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-29T15:55:43.255 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:43.256+0000 7f8a46c39d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-11-29T15:55:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:55:43.256+0000 7f8a46c39d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-11-29T15:55:45.417 INFO:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-11-29T15:55:45.474 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-29T15:55:45.475 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:45.476 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-29T15:55:45.523 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-11-29T15:55:45.532 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2020-11-29T15:55:45.532 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB4xMNf4UE5GhAAFH/8eJKI16xt9J163JdlWA== 2020-11-29T15:55:45.533 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:45.533 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:45.533 INFO:teuthology.orchestra.run.smithi184.stderr:56 bytes copied, 0.000210894 s, 266 kB/s 2020-11-29T15:55:45.534 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:45.535 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-29T15:55:45.579 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-11-29T15:55:45.587 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2020-11-29T15:55:45.587 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB6xMNfsTy8JxAAt/b9ZAfgNKC6cykt8XZHWg== 2020-11-29T15:55:45.588 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:45.588 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:45.588 INFO:teuthology.orchestra.run.smithi184.stderr:56 bytes copied, 0.000212393 s, 264 kB/s 2020-11-29T15:55:45.589 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:45.589 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-29T15:55:45.635 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-11-29T15:55:45.644 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2020-11-29T15:55:45.644 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB8xMNfOBqJOBAAUI55uOh8QSfQFKEcZ6M8JQ== 2020-11-29T15:55:45.644 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:45.645 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:45.645 INFO:teuthology.orchestra.run.smithi184.stderr:56 bytes copied, 0.000208781 s, 268 kB/s 2020-11-29T15:55:45.647 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:45.647 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-29T15:55:45.695 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-11-29T15:55:45.703 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2020-11-29T15:55:45.704 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB/xMNfsZFgDxAAOjy+glSJ+BsHucO6m1gEXg== 2020-11-29T15:55:45.704 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:45.704 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:45.704 INFO:teuthology.orchestra.run.smithi184.stderr:56 bytes copied, 9.944e-05 s, 563 kB/s 2020-11-29T15:55:45.705 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:45.706 INFO:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-29T15:55:45.751 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-11-29T15:55:45.752 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2020-11-29T15:55:45.752 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB4xMNfHJnaHhAA+c+iwYIxbNxJHEFZ0v8zKg== 2020-11-29T15:55:45.753 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:45.753 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:45.753 INFO:teuthology.orchestra.run.smithi184.stderr:59 bytes copied, 0.00016924 s, 349 kB/s 2020-11-29T15:55:45.754 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:45.754 INFO:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-29T15:55:45.800 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-11-29T15:55:45.802 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2020-11-29T15:55:45.802 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB4xMNfZYYMJBAAW/GNygjtNvkcoyb+7Llh8Q== 2020-11-29T15:55:45.803 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:45.803 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:45.803 INFO:teuthology.orchestra.run.smithi184.stderr:59 bytes copied, 0.00015215 s, 388 kB/s 2020-11-29T15:55:45.804 INFO:tasks.ceph:Adding keys to all mons... 2020-11-29T15:55:45.804 INFO:teuthology.orchestra.run.smithi184:> sudo tee -a /etc/ceph/c2.keyring 2020-11-29T15:55:45.855 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2020-11-29T15:55:45.856 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB4xMNf4UE5GhAAFH/8eJKI16xt9J163JdlWA== 2020-11-29T15:55:45.856 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2020-11-29T15:55:45.856 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB6xMNfsTy8JxAAt/b9ZAfgNKC6cykt8XZHWg== 2020-11-29T15:55:45.856 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2020-11-29T15:55:45.857 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB8xMNfOBqJOBAAUI55uOh8QSfQFKEcZ6M8JQ== 2020-11-29T15:55:45.857 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2020-11-29T15:55:45.857 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB/xMNfsZFgDxAAOjy+glSJ+BsHucO6m1gEXg== 2020-11-29T15:55:45.857 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2020-11-29T15:55:45.858 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB4xMNfHJnaHhAA+c+iwYIxbNxJHEFZ0v8zKg== 2020-11-29T15:55:45.858 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2020-11-29T15:55:45.858 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQB4xMNfZYYMJBAAW/GNygjtNvkcoyb+7Llh8Q== 2020-11-29T15:55:45.859 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-29T15:55:45.943 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-29T15:55:46.028 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-29T15:55:46.112 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-29T15:55:46.196 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-29T15:55:46.282 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-29T15:55:46.371 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-29T15:55:46.371 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-11-29T15:55:46.429 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-11-29T15:55:46.524 INFO:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-11-29T15:55:46.540 INFO:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-11-29T15:55:46.588 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-11-29T15:55:46.589 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-11-29T15:55:46.589 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-11-29T15:55:46.632 INFO:tasks.ceph.c2.mon.a:Started 2020-11-29T15:55:46.633 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-11-29T15:55:46.633 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-11-29T15:55:46.634 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-11-29T15:55:46.636 INFO:tasks.ceph.c2.mgr.x:Started 2020-11-29T15:55:46.636 INFO:tasks.ceph:Setting crush tunables to default 2020-11-29T15:55:46.637 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd crush tunables default 2020-11-29T15:55:47.110 INFO:teuthology.orchestra.run.smithi184.stderr:adjusted tunables profile to default 2020-11-29T15:55:47.121 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-11-29T15:55:47.122 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:47.122 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-29T15:55:47.127 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-11-29T15:55:47.133 INFO:teuthology.orchestra.run.smithi184.stdout:7c0ec2b4-1a99-4ae1-b732-22e5afa61367 2020-11-29T15:55:47.133 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:47.134 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:47.134 INFO:teuthology.orchestra.run.smithi184.stderr:37 bytes copied, 8.285e-05 s, 447 kB/s 2020-11-29T15:55:47.135 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:47.135 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-29T15:55:47.183 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-11-29T15:55:47.192 INFO:teuthology.orchestra.run.smithi184.stdout:72dedbda-a383-4521-a41b-3b80bc06c856 2020-11-29T15:55:47.192 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:47.192 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:47.193 INFO:teuthology.orchestra.run.smithi184.stderr:37 bytes copied, 0.000188004 s, 197 kB/s 2020-11-29T15:55:47.194 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-11-29T15:55:47.194 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-29T15:55:47.238 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-11-29T15:55:47.246 INFO:teuthology.orchestra.run.smithi184.stdout:4e13cd26-ac47-42c4-9da4-9d9e8aae2fba 2020-11-29T15:55:47.246 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-11-29T15:55:47.247 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-11-29T15:55:47.247 INFO:teuthology.orchestra.run.smithi184.stderr:37 bytes copied, 0.000209157 s, 177 kB/s 2020-11-29T15:55:47.248 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new 7c0ec2b4-1a99-4ae1-b732-22e5afa61367 0 2020-11-29T15:55:47.506 INFO:teuthology.orchestra.run.smithi184.stdout:0 2020-11-29T15:55:47.518 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new 72dedbda-a383-4521-a41b-3b80bc06c856 1 2020-11-29T15:55:47.728 INFO:teuthology.orchestra.run.smithi184.stdout:1 2020-11-29T15:55:47.745 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new 4e13cd26-ac47-42c4-9da4-9d9e8aae2fba 2 2020-11-29T15:55:47.956 INFO:teuthology.orchestra.run.smithi184.stdout:2 2020-11-29T15:55:47.969 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-11-29T15:55:47.970 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-11-29T15:55:47.973 INFO:tasks.ceph.c2.osd.0:Started 2020-11-29T15:55:47.974 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-11-29T15:55:47.974 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-11-29T15:55:47.977 INFO:tasks.ceph.c2.osd.1:Started 2020-11-29T15:55:47.978 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-11-29T15:55:47.979 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-11-29T15:55:47.982 INFO:tasks.ceph.c2.osd.2:Started 2020-11-29T15:55:47.983 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-29T15:55:48.286 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-29T15:55:48.332 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2020-11-29T15:55:48.336+0000 7fd8fdfc7d80 -1 Falling back to public interface 2020-11-29T15:55:48.484 INFO:teuthology.misc.health.smithi184.stdout: 2020-11-29T15:55:48.484 INFO:teuthology.misc.health.smithi184.stdout:{"epoch":5,"fsid":"ebcec895-1108-492d-9d57-61e3df7f12c1","created":"2020-11-29T15:55:46.755178+0000","modified":"2020-11-29T15:55:47.958461+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"7c0ec2b4-1a99-4ae1-b732-22e5afa61367","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"72dedbda-a383-4521-a41b-3b80bc06c856","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e13cd26-ac47-42c4-9da4-9d9e8aae2fba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:48.494 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-11-29T15:55:48.580 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2020-11-29T15:55:48.584+0000 7fd6438e5d80 -1 Falling back to public interface 2020-11-29T15:55:48.581 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2020-11-29T15:55:48.584+0000 7f4442842d80 -1 Falling back to public interface 2020-11-29T15:55:48.943 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2020-11-29T15:55:48.948+0000 7fd6438e5d80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-29T15:55:48.985 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2020-11-29T15:55:48.980+0000 7fd8fdfc7d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-29T15:55:48.989 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2020-11-29T15:55:48.992+0000 7f4442842d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-29T15:55:50.784 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2020-11-29T15:55:50.788+0000 7fd8f6c94700 -1 osd.0 0 waiting for initial osdmap 2020-11-29T15:55:50.784 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2020-11-29T15:55:50.788+0000 7f443b50f700 -1 osd.1 0 waiting for initial osdmap 2020-11-29T15:55:50.785 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2020-11-29T15:55:50.788+0000 7fd63adaf700 -1 osd.2 0 waiting for initial osdmap 2020-11-29T15:55:54.798 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-11-29T15:55:55.080 INFO:teuthology.misc.health.smithi184.stdout: 2020-11-29T15:55:55.080 INFO:teuthology.misc.health.smithi184.stdout:{"epoch":10,"fsid":"ebcec895-1108-492d-9d57-61e3df7f12c1","created":"2020-11-29T15:55:46.755178+0000","modified":"2020-11-29T15:55:53.797249+0000","last_up_change":"2020-11-29T15:55:51.783099+0000","last_in_change":"2020-11-29T15:55:51.783099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-29T15:55:50.597401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"7c0ec2b4-1a99-4ae1-b732-22e5afa61367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":12578},{"type":"v1","addr":"172.21.15.184:6801","nonce":12578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":12578},{"type":"v1","addr":"172.21.15.184:6803","nonce":12578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":12578},{"type":"v1","addr":"172.21.15.184:6807","nonce":12578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":12578},{"type":"v1","addr":"172.21.15.184:6805","nonce":12578}]},"public_addr":"172.21.15.184:6801/12578","cluster_addr":"172.21.15.184:6803/12578","heartbeat_back_addr":"172.21.15.184:6807/12578","heartbeat_front_addr":"172.21.15.184:6805/12578","state":["exists","up"]},{"osd":1,"uuid":"72dedbda-a383-4521-a41b-3b80bc06c856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":12580},{"type":"v1","addr":"172.21.15.184:6815","nonce":12580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":12580},{"type":"v1","addr":"172.21.15.184:6819","nonce":12580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":12580},{"type":"v1","addr":"172.21.15.184:6823","nonce":12580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":12580},{"type":"v1","addr":"172.21.15.184:6821","nonce":12580}]},"public_addr":"172.21.15.184:6815/12580","cluster_addr":"172.21.15.184:6819/12580","heartbeat_back_addr":"172.21.15.184:6823/12580","heartbeat_front_addr":"172.21.15.184:6821/12580","state":["exists","up"]},{"osd":2,"uuid":"4e13cd26-ac47-42c4-9da4-9d9e8aae2fba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":12579},{"type":"v1","addr":"172.21.15.184:6809","nonce":12579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":12579},{"type":"v1","addr":"172.21.15.184:6811","nonce":12579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":12579},{"type":"v1","addr":"172.21.15.184:6817","nonce":12579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6813","nonce":12579},{"type":"v1","addr":"172.21.15.184:6814","nonce":12579}]},"public_addr":"172.21.15.184:6809/12579","cluster_addr":"172.21.15.184:6811/12579","heartbeat_back_addr":"172.21.15.184:6817/12579","heartbeat_front_addr":"172.21.15.184:6814/12579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:49.990483+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:50.005214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:49.926585+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:55.093 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-11-29T15:55:55.094 INFO:tasks.ceph:Creating RBD pool 2020-11-29T15:55:55.094 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd pool create rbd 8 2020-11-29T15:55:55.828 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'rbd' created 2020-11-29T15:55:55.842 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-29T15:55:56.841 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-29T15:55:56.854 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-11-29T15:55:56.855 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-29T15:55:56.856 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-29T15:55:57.152 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:57.153 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":12,"fsid":"ebcec895-1108-492d-9d57-61e3df7f12c1","created":"2020-11-29T15:55:46.755178+0000","modified":"2020-11-29T15:55:56.821152+0000","last_up_change":"2020-11-29T15:55:51.783099+0000","last_in_change":"2020-11-29T15:55:51.783099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-29T15:55:50.597401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-29T15:55:55.388387+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7c0ec2b4-1a99-4ae1-b732-22e5afa61367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":12578},{"type":"v1","addr":"172.21.15.184:6801","nonce":12578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":12578},{"type":"v1","addr":"172.21.15.184:6803","nonce":12578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":12578},{"type":"v1","addr":"172.21.15.184:6807","nonce":12578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":12578},{"type":"v1","addr":"172.21.15.184:6805","nonce":12578}]},"public_addr":"172.21.15.184:6801/12578","cluster_addr":"172.21.15.184:6803/12578","heartbeat_back_addr":"172.21.15.184:6807/12578","heartbeat_front_addr":"172.21.15.184:6805/12578","state":["exists","up"]},{"osd":1,"uuid":"72dedbda-a383-4521-a41b-3b80bc06c856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":12580},{"type":"v1","addr":"172.21.15.184:6815","nonce":12580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":12580},{"type":"v1","addr":"172.21.15.184:6819","nonce":12580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":12580},{"type":"v1","addr":"172.21.15.184:6823","nonce":12580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":12580},{"type":"v1","addr":"172.21.15.184:6821","nonce":12580}]},"public_addr":"172.21.15.184:6815/12580","cluster_addr":"172.21.15.184:6819/12580","heartbeat_back_addr":"172.21.15.184:6823/12580","heartbeat_front_addr":"172.21.15.184:6821/12580","state":["exists","up"]},{"osd":2,"uuid":"4e13cd26-ac47-42c4-9da4-9d9e8aae2fba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":12579},{"type":"v1","addr":"172.21.15.184:6809","nonce":12579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":12579},{"type":"v1","addr":"172.21.15.184:6811","nonce":12579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":12579},{"type":"v1","addr":"172.21.15.184:6817","nonce":12579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6813","nonce":12579},{"type":"v1","addr":"172.21.15.184:6814","nonce":12579}]},"public_addr":"172.21.15.184:6809/12579","cluster_addr":"172.21.15.184:6811/12579","heartbeat_back_addr":"172.21.15.184:6817/12579","heartbeat_front_addr":"172.21.15.184:6814/12579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:49.990483+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:50.005214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:49.926585+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:57.165 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-29T15:55:50.597401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-29T15:55:55.388387+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-29T15:55:57.166 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2020-11-29T15:55:57.456 INFO:teuthology.orchestra.run.smithi184.stdout:pg_num: 1 2020-11-29T15:55:57.474 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2020-11-29T15:55:57.767 INFO:teuthology.orchestra.run.smithi184.stdout:pg_num: 8 2020-11-29T15:55:57.781 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-11-29T15:55:57.782 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-11-29T15:55:57.782 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2020-11-29T15:55:58.112 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:58.126 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":12395},{"type":"v1","addr":"172.21.15.184:6825","nonce":12395}]},"active_addr":"172.21.15.184:6825/12395","active_change":"2020-11-29T15:55:50.570309+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":3647704408}]},{"addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":667556387}]},{"addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":1012371477}]}]}} 2020-11-29T15:55:58.128 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-11-29T15:55:58.128 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-11-29T15:55:58.129 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-29T15:55:58.414 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:58.415 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":13,"fsid":"ebcec895-1108-492d-9d57-61e3df7f12c1","created":"2020-11-29T15:55:46.755178+0000","modified":"2020-11-29T15:55:57.828102+0000","last_up_change":"2020-11-29T15:55:51.783099+0000","last_in_change":"2020-11-29T15:55:51.783099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-29T15:55:50.597401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-29T15:55:55.388387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7c0ec2b4-1a99-4ae1-b732-22e5afa61367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":12578},{"type":"v1","addr":"172.21.15.184:6801","nonce":12578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":12578},{"type":"v1","addr":"172.21.15.184:6803","nonce":12578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":12578},{"type":"v1","addr":"172.21.15.184:6807","nonce":12578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":12578},{"type":"v1","addr":"172.21.15.184:6805","nonce":12578}]},"public_addr":"172.21.15.184:6801/12578","cluster_addr":"172.21.15.184:6803/12578","heartbeat_back_addr":"172.21.15.184:6807/12578","heartbeat_front_addr":"172.21.15.184:6805/12578","state":["exists","up"]},{"osd":1,"uuid":"72dedbda-a383-4521-a41b-3b80bc06c856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":12580},{"type":"v1","addr":"172.21.15.184:6815","nonce":12580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":12580},{"type":"v1","addr":"172.21.15.184:6819","nonce":12580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":12580},{"type":"v1","addr":"172.21.15.184:6823","nonce":12580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":12580},{"type":"v1","addr":"172.21.15.184:6821","nonce":12580}]},"public_addr":"172.21.15.184:6815/12580","cluster_addr":"172.21.15.184:6819/12580","heartbeat_back_addr":"172.21.15.184:6823/12580","heartbeat_front_addr":"172.21.15.184:6821/12580","state":["exists","up"]},{"osd":2,"uuid":"4e13cd26-ac47-42c4-9da4-9d9e8aae2fba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":12579},{"type":"v1","addr":"172.21.15.184:6809","nonce":12579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":12579},{"type":"v1","addr":"172.21.15.184:6811","nonce":12579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":12579},{"type":"v1","addr":"172.21.15.184:6817","nonce":12579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6813","nonce":12579},{"type":"v1","addr":"172.21.15.184:6814","nonce":12579}]},"public_addr":"172.21.15.184:6809/12579","cluster_addr":"172.21.15.184:6811/12579","heartbeat_back_addr":"172.21.15.184:6817/12579","heartbeat_front_addr":"172.21.15.184:6814/12579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:49.990483+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:50.005214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:49.926585+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:58.429 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-11-29T15:55:58.430 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2020-11-29T15:55:58.717 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:55:58.717 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":13,"fsid":"ebcec895-1108-492d-9d57-61e3df7f12c1","created":"2020-11-29T15:55:46.755178+0000","modified":"2020-11-29T15:55:57.828102+0000","last_up_change":"2020-11-29T15:55:51.783099+0000","last_in_change":"2020-11-29T15:55:51.783099+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-29T15:55:50.597401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-29T15:55:55.388387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7c0ec2b4-1a99-4ae1-b732-22e5afa61367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":12578},{"type":"v1","addr":"172.21.15.184:6801","nonce":12578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":12578},{"type":"v1","addr":"172.21.15.184:6803","nonce":12578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":12578},{"type":"v1","addr":"172.21.15.184:6807","nonce":12578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":12578},{"type":"v1","addr":"172.21.15.184:6805","nonce":12578}]},"public_addr":"172.21.15.184:6801/12578","cluster_addr":"172.21.15.184:6803/12578","heartbeat_back_addr":"172.21.15.184:6807/12578","heartbeat_front_addr":"172.21.15.184:6805/12578","state":["exists","up"]},{"osd":1,"uuid":"72dedbda-a383-4521-a41b-3b80bc06c856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":12580},{"type":"v1","addr":"172.21.15.184:6815","nonce":12580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":12580},{"type":"v1","addr":"172.21.15.184:6819","nonce":12580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":12580},{"type":"v1","addr":"172.21.15.184:6823","nonce":12580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":12580},{"type":"v1","addr":"172.21.15.184:6821","nonce":12580}]},"public_addr":"172.21.15.184:6815/12580","cluster_addr":"172.21.15.184:6819/12580","heartbeat_back_addr":"172.21.15.184:6823/12580","heartbeat_front_addr":"172.21.15.184:6821/12580","state":["exists","up"]},{"osd":2,"uuid":"4e13cd26-ac47-42c4-9da4-9d9e8aae2fba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":12579},{"type":"v1","addr":"172.21.15.184:6809","nonce":12579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":12579},{"type":"v1","addr":"172.21.15.184:6811","nonce":12579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":12579},{"type":"v1","addr":"172.21.15.184:6817","nonce":12579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6813","nonce":12579},{"type":"v1","addr":"172.21.15.184:6814","nonce":12579}]},"public_addr":"172.21.15.184:6809/12579","cluster_addr":"172.21.15.184:6811/12579","heartbeat_back_addr":"172.21.15.184:6817/12579","heartbeat_front_addr":"172.21.15.184:6814/12579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:49.990483+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:50.005214+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-29T15:55:49.926585+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-29T15:55:58.732 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2020-11-29T15:55:58.897 INFO:teuthology.orchestra.run.smithi184.stdout:34359738372 2020-11-29T15:55:58.898 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2020-11-29T15:55:59.065 INFO:teuthology.orchestra.run.smithi184.stdout:34359738372 2020-11-29T15:55:59.066 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2020-11-29T15:55:59.240 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2020-11-29T15:55:59.240 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-29T15:55:59.538 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2020-11-29T15:55:59.552 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-11-29T15:56:00.553 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2020-11-29T15:56:00.854 INFO:teuthology.orchestra.run.smithi184.stdout:34359738372 2020-11-29T15:56:00.872 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2020-11-29T15:56:00.873 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2020-11-29T15:56:01.166 INFO:teuthology.orchestra.run.smithi184.stdout:34359738372 2020-11-29T15:56:01.186 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2020-11-29T15:56:01.186 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2020-11-29T15:56:01.479 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2020-11-29T15:56:01.497 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2020-11-29T15:56:01.498 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-11-29T15:56:01.498 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-29T15:56:01.782 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:56:01.785 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2020-11-29T15:56:01.796 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-11-29T15:56:00.580213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.749840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.846864+0000","last_change":"2020-11-29T15:55:56.844722+0000","last_active":"2020-11-29T15:55:56.846864+0000","last_peered":"2020-11-29T15:55:56.846864+0000","last_clean":"2020-11-29T15:55:56.846864+0000","last_became_active":"2020-11-29T15:55:56.844178+0000","last_became_peered":"2020-11-29T15:55:56.844178+0000","last_unstale":"2020-11-29T15:55:56.846864+0000","last_undegraded":"2020-11-29T15:55:56.846864+0000","last_fullsized":"2020-11-29T15:55:56.846864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.847111+0000","last_change":"2020-11-29T15:55:56.845653+0000","last_active":"2020-11-29T15:55:56.847111+0000","last_peered":"2020-11-29T15:55:56.847111+0000","last_clean":"2020-11-29T15:55:56.847111+0000","last_became_active":"2020-11-29T15:55:56.845216+0000","last_became_peered":"2020-11-29T15:55:56.845216+0000","last_unstale":"2020-11-29T15:55:56.847111+0000","last_undegraded":"2020-11-29T15:55:56.847111+0000","last_fullsized":"2020-11-29T15:55:56.847111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.846826+0000","last_change":"2020-11-29T15:55:56.845064+0000","last_active":"2020-11-29T15:55:56.846826+0000","last_peered":"2020-11-29T15:55:56.846826+0000","last_clean":"2020-11-29T15:55:56.846826+0000","last_became_active":"2020-11-29T15:55:56.844500+0000","last_became_peered":"2020-11-29T15:55:56.844500+0000","last_unstale":"2020-11-29T15:55:56.846826+0000","last_undegraded":"2020-11-29T15:55:56.846826+0000","last_fullsized":"2020-11-29T15:55:56.846826+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.840904+0000","last_change":"2020-11-29T15:55:56.839836+0000","last_active":"2020-11-29T15:55:56.840904+0000","last_peered":"2020-11-29T15:55:56.840904+0000","last_clean":"2020-11-29T15:55:56.840904+0000","last_became_active":"2020-11-29T15:55:56.839103+0000","last_became_peered":"2020-11-29T15:55:56.839103+0000","last_unstale":"2020-11-29T15:55:56.840904+0000","last_undegraded":"2020-11-29T15:55:56.840904+0000","last_fullsized":"2020-11-29T15:55:56.840904+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.837434+0000","last_change":"2020-11-29T15:55:53.048066+0000","last_active":"2020-11-29T15:55:56.837434+0000","last_peered":"2020-11-29T15:55:56.837434+0000","last_clean":"2020-11-29T15:55:56.837434+0000","last_became_active":"2020-11-29T15:55:53.046492+0000","last_became_peered":"2020-11-29T15:55:53.046492+0000","last_unstale":"2020-11-29T15:55:56.837434+0000","last_undegraded":"2020-11-29T15:55:56.837434+0000","last_fullsized":"2020-11-29T15:55:56.837434+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:50.774615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:50.774615+0000","last_clean_scrub_stamp":"2020-11-29T15:55:50.774615+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.844918+0000","last_change":"2020-11-29T15:55:56.843841+0000","last_active":"2020-11-29T15:55:56.844918+0000","last_peered":"2020-11-29T15:55:56.844918+0000","last_clean":"2020-11-29T15:55:56.844918+0000","last_became_active":"2020-11-29T15:55:56.843373+0000","last_became_peered":"2020-11-29T15:55:56.843373+0000","last_unstale":"2020-11-29T15:55:56.844918+0000","last_undegraded":"2020-11-29T15:55:56.844918+0000","last_fullsized":"2020-11-29T15:55:56.844918+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.840877+0000","last_change":"2020-11-29T15:55:56.839941+0000","last_active":"2020-11-29T15:55:56.840877+0000","last_peered":"2020-11-29T15:55:56.840877+0000","last_clean":"2020-11-29T15:55:56.840877+0000","last_became_active":"2020-11-29T15:55:56.839365+0000","last_became_peered":"2020-11-29T15:55:56.839365+0000","last_unstale":"2020-11-29T15:55:56.840877+0000","last_undegraded":"2020-11-29T15:55:56.840877+0000","last_fullsized":"2020-11-29T15:55:56.840877+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.846016+0000","last_change":"2020-11-29T15:55:56.844454+0000","last_active":"2020-11-29T15:55:56.846016+0000","last_peered":"2020-11-29T15:55:56.846016+0000","last_clean":"2020-11-29T15:55:56.846016+0000","last_became_active":"2020-11-29T15:55:56.843643+0000","last_became_peered":"2020-11-29T15:55:56.843643+0000","last_unstale":"2020-11-29T15:55:56.846016+0000","last_undegraded":"2020-11-29T15:55:56.846016+0000","last_fullsized":"2020-11-29T15:55:56.846016+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.845066+0000","last_change":"2020-11-29T15:55:56.842897+0000","last_active":"2020-11-29T15:55:56.845066+0000","last_peered":"2020-11-29T15:55:56.845066+0000","last_clean":"2020-11-29T15:55:56.845066+0000","last_became_active":"2020-11-29T15:55:56.842575+0000","last_became_peered":"2020-11-29T15:55:56.842575+0000","last_unstale":"2020-11-29T15:55:56.845066+0000","last_undegraded":"2020-11-29T15:55:56.845066+0000","last_fullsized":"2020-11-29T15:55:56.845066+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.474}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0779999999999998}]}]}]}} 2020-11-29T15:56:01.797 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2020-11-29T15:56:02.075 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:56:02.077 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2020-11-29T15:56:02.089 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-11-29T15:56:00.580213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.749840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.846864+0000","last_change":"2020-11-29T15:55:56.844722+0000","last_active":"2020-11-29T15:55:56.846864+0000","last_peered":"2020-11-29T15:55:56.846864+0000","last_clean":"2020-11-29T15:55:56.846864+0000","last_became_active":"2020-11-29T15:55:56.844178+0000","last_became_peered":"2020-11-29T15:55:56.844178+0000","last_unstale":"2020-11-29T15:55:56.846864+0000","last_undegraded":"2020-11-29T15:55:56.846864+0000","last_fullsized":"2020-11-29T15:55:56.846864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.847111+0000","last_change":"2020-11-29T15:55:56.845653+0000","last_active":"2020-11-29T15:55:56.847111+0000","last_peered":"2020-11-29T15:55:56.847111+0000","last_clean":"2020-11-29T15:55:56.847111+0000","last_became_active":"2020-11-29T15:55:56.845216+0000","last_became_peered":"2020-11-29T15:55:56.845216+0000","last_unstale":"2020-11-29T15:55:56.847111+0000","last_undegraded":"2020-11-29T15:55:56.847111+0000","last_fullsized":"2020-11-29T15:55:56.847111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.846826+0000","last_change":"2020-11-29T15:55:56.845064+0000","last_active":"2020-11-29T15:55:56.846826+0000","last_peered":"2020-11-29T15:55:56.846826+0000","last_clean":"2020-11-29T15:55:56.846826+0000","last_became_active":"2020-11-29T15:55:56.844500+0000","last_became_peered":"2020-11-29T15:55:56.844500+0000","last_unstale":"2020-11-29T15:55:56.846826+0000","last_undegraded":"2020-11-29T15:55:56.846826+0000","last_fullsized":"2020-11-29T15:55:56.846826+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.840904+0000","last_change":"2020-11-29T15:55:56.839836+0000","last_active":"2020-11-29T15:55:56.840904+0000","last_peered":"2020-11-29T15:55:56.840904+0000","last_clean":"2020-11-29T15:55:56.840904+0000","last_became_active":"2020-11-29T15:55:56.839103+0000","last_became_peered":"2020-11-29T15:55:56.839103+0000","last_unstale":"2020-11-29T15:55:56.840904+0000","last_undegraded":"2020-11-29T15:55:56.840904+0000","last_fullsized":"2020-11-29T15:55:56.840904+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.837434+0000","last_change":"2020-11-29T15:55:53.048066+0000","last_active":"2020-11-29T15:55:56.837434+0000","last_peered":"2020-11-29T15:55:56.837434+0000","last_clean":"2020-11-29T15:55:56.837434+0000","last_became_active":"2020-11-29T15:55:53.046492+0000","last_became_peered":"2020-11-29T15:55:53.046492+0000","last_unstale":"2020-11-29T15:55:56.837434+0000","last_undegraded":"2020-11-29T15:55:56.837434+0000","last_fullsized":"2020-11-29T15:55:56.837434+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:50.774615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:50.774615+0000","last_clean_scrub_stamp":"2020-11-29T15:55:50.774615+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.844918+0000","last_change":"2020-11-29T15:55:56.843841+0000","last_active":"2020-11-29T15:55:56.844918+0000","last_peered":"2020-11-29T15:55:56.844918+0000","last_clean":"2020-11-29T15:55:56.844918+0000","last_became_active":"2020-11-29T15:55:56.843373+0000","last_became_peered":"2020-11-29T15:55:56.843373+0000","last_unstale":"2020-11-29T15:55:56.844918+0000","last_undegraded":"2020-11-29T15:55:56.844918+0000","last_fullsized":"2020-11-29T15:55:56.844918+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.840877+0000","last_change":"2020-11-29T15:55:56.839941+0000","last_active":"2020-11-29T15:55:56.840877+0000","last_peered":"2020-11-29T15:55:56.840877+0000","last_clean":"2020-11-29T15:55:56.840877+0000","last_became_active":"2020-11-29T15:55:56.839365+0000","last_became_peered":"2020-11-29T15:55:56.839365+0000","last_unstale":"2020-11-29T15:55:56.840877+0000","last_undegraded":"2020-11-29T15:55:56.840877+0000","last_fullsized":"2020-11-29T15:55:56.840877+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.846016+0000","last_change":"2020-11-29T15:55:56.844454+0000","last_active":"2020-11-29T15:55:56.846016+0000","last_peered":"2020-11-29T15:55:56.846016+0000","last_clean":"2020-11-29T15:55:56.846016+0000","last_became_active":"2020-11-29T15:55:56.843643+0000","last_became_peered":"2020-11-29T15:55:56.843643+0000","last_unstale":"2020-11-29T15:55:56.846016+0000","last_undegraded":"2020-11-29T15:55:56.846016+0000","last_fullsized":"2020-11-29T15:55:56.846016+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-11-29T15:55:56.845066+0000","last_change":"2020-11-29T15:55:56.842897+0000","last_active":"2020-11-29T15:55:56.845066+0000","last_peered":"2020-11-29T15:55:56.845066+0000","last_clean":"2020-11-29T15:55:56.845066+0000","last_became_active":"2020-11-29T15:55:56.842575+0000","last_became_peered":"2020-11-29T15:55:56.842575+0000","last_unstale":"2020-11-29T15:55:56.845066+0000","last_undegraded":"2020-11-29T15:55:56.845066+0000","last_fullsized":"2020-11-29T15:55:56.845066+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-29T15:55:55.813075+0000","last_clean_scrub_stamp":"2020-11-29T15:55:55.813075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.474}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0779999999999998}]}]}]}} 2020-11-29T15:56:02.090 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-11-29T15:56:02.090 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-11-29T15:56:02.091 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-11-29T15:56:02.091 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2020-11-29T15:56:02.423 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T15:56:02.437 INFO:teuthology.orchestra.run.smithi184.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-29T15:56:02.437 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-11-29T15:56:02.438 INFO:teuthology.run_tasks:Running task rgw... 2020-11-29T15:56:02.455 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-11-29T15:56:02.456 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-11-29T15:56:02.456 INFO:tasks.rgw:Creating data pools 2020-11-29T15:56:02.456 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-11-29T15:56:02.457 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-29T15:56:03.241 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'default.rgw.buckets.data' created 2020-11-29T15:56:03.265 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-29T15:56:04.271 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-29T15:56:04.290 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-29T15:56:05.272 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'default.rgw.buckets.index' created 2020-11-29T15:56:05.288 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-29T15:56:06.293 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-29T15:56:06.311 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-11-29T15:56:06.312 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-11-29T15:56:06.600 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-29T15:56:06.611 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-11-29T15:56:07.272 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-29T15:56:07.286 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-11-29T15:56:07.580 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-29T15:56:07.597 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-11-29T15:56:08.281 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-29T15:56:08.294 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-11-29T15:56:08.295 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-29T15:56:08.754 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.data' created 2020-11-29T15:56:08.772 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-29T15:56:09.768 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-29T15:56:09.791 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-29T15:56:10.781 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.index' created 2020-11-29T15:56:10.796 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-29T15:56:11.780 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-29T15:56:11.800 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-11-29T15:56:11.801 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-11-29T15:56:12.092 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.data' already exists 2020-11-29T15:56:12.104 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-11-29T15:56:12.778 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-11-29T15:56:12.792 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-11-29T15:56:13.082 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.index' already exists 2020-11-29T15:56:13.093 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-11-29T15:56:13.779 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-11-29T15:56:13.791 DEBUG:tasks.rgw:Pools created 2020-11-29T15:56:13.792 INFO:tasks.rgw:Configuring compression type = random 2020-11-29T15:56:13.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-11-29T15:56:13.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-11-29T15:56:13.793 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-11-29T15:56:13.829 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:13.829 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:13.844 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:13.844+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.218 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.220+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.218 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.220+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.226 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.228+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.226 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.228+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.227 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.228+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.228 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.228+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.228 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.232+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.247 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.248+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.248 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.248+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.248 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.248+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.248 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.248 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.249 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.250 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 10 cannot find current period zonegroup using local zonegroup 2020-11-29T15:56:15.250 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.250 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.250 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.251 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.251 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 10 Creating default zonegroup 2020-11-29T15:56:15.251 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 20 get_system_obj_state: rctx=0x7ffc7a94edf0 obj=:.pools.avail state=0x55680f108a30 s->prefetch_data=0 2020-11-29T15:56:15.252 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.252+0000 7fe6a103e080 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-29T15:56:15.271 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.272+0000 7fe6a103e080 10 WARNING: store->list_zones() returned r=-2 2020-11-29T15:56:15.271 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.272+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.272 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.272+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.280 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.284+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.281 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.284+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.281 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.284+0000 7fe6a103e080 10 could not read realm id: (2) No such file or directory 2020-11-29T15:56:15.282 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.284+0000 7fe6a103e080 10 WARNING: failed to set zone as default, r=-22 2020-11-29T15:56:15.282 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.284+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.282 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.284+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.288 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.289 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 20 rados_obj.operate() r=0 bl.length=873 2020-11-29T15:56:15.289 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.290 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:15.291 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 20 zonegroup default 2020-11-29T15:56:15.291 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.291 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:15.292 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 10 Cannot find current period zone using local zone 2020-11-29T15:56:15.292 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 10 Using default name default 2020-11-29T15:56:15.292 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.292+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.293 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.296+0000 7fe6a103e080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:15.293 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.296+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:15.294 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.296+0000 7fe6a103e080 20 rados_obj.operate() r=0 bl.length=873 2020-11-29T15:56:15.294 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.296+0000 7fe6a103e080 20 zone default found 2020-11-29T15:56:15.294 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:15.296+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:17.240 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:17.240+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:17.240 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:17.240+0000 7fe6a103e080 20 rados->read ofs=0 len=0 2020-11-29T15:56:17.243 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:17.244+0000 7fe6a103e080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:17.243 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:17.244+0000 7fe6a103e080 20 started sync module instance, tier type = 2020-11-29T15:56:17.244 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:17.244+0000 7fe6a103e080 20 started zone id=a76bf446-de9f-465c-9e33-558473330d4b (name=default) with tier type = 2020-11-29T15:56:19.299 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.300+0000 7fe6a103e080 20 add_watcher() i=0 2020-11-29T15:56:19.299 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.300+0000 7fe6a103e080 20 add_watcher() i=1 2020-11-29T15:56:19.299 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.300+0000 7fe6a103e080 20 add_watcher() i=2 2020-11-29T15:56:19.300 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.300+0000 7fe6a103e080 20 add_watcher() i=3 2020-11-29T15:56:19.300 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.300+0000 7fe6a103e080 20 add_watcher() i=4 2020-11-29T15:56:19.300 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.300+0000 7fe6a103e080 20 add_watcher() i=5 2020-11-29T15:56:19.300 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.300+0000 7fe6a103e080 20 add_watcher() i=6 2020-11-29T15:56:19.301 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.304+0000 7fe6a103e080 20 add_watcher() i=7 2020-11-29T15:56:19.301 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.304+0000 7fe6a103e080 2 all 8 watchers are set, enabling cache 2020-11-29T15:56:19.303 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.304+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:19.304 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.304+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:19.304 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.308+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:19.305 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.308+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:19.305 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.308+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:19.308 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.312+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:19.309 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.312+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:19.313 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.316+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:19.314 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.316+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:19.317 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.320+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:19.320 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.320+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:19.322 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.324+0000 7fe6a103e080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:19.324 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.328+0000 7fe65d7f2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-29T15:56:19.324 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.328+0000 7fe6a103e080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-29T15:56:19.324 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:19.328+0000 7fe6a103e080 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-29T15:56:23.261 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.264+0000 7fe6a103e080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-29T15:56:23.383 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.384+0000 7fe5b37e6700 20 reqs_thread_entry: start 2020-11-29T15:56:23.383 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.384+0000 7fe5b17e2700 20 reqs_thread_entry: start 2020-11-29T15:56:23.383 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.384+0000 7fe6a103e080 20 init_complete bucket index max shards: 11 2020-11-29T15:56:23.383 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.384+0000 7fe6a103e080 10 Started notification manager with: 1 workers 2020-11-29T15:56:23.384 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.384+0000 7fe59b7fe700 20 INFO: next queues processing will happen at: Sun Nov 29 15:56:53 2020 2020-11-29T15:56:23.384 INFO:teuthology.orchestra.run.smithi168.stderr: 2020-11-29T15:56:23.384 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.384+0000 7fe59affd700 20 reqs_thread_entry: start 2020-11-29T15:56:23.410 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T15:56:23.411 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.412+0000 7fe6a103e080 20 remove_watcher() i=0 2020-11-29T15:56:23.411 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.412+0000 7fe6a103e080 2 removed watcher, disabling cache 2020-11-29T15:56:23.413 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.416+0000 7fe6a103e080 20 remove_watcher() i=1 2020-11-29T15:56:23.416 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.420+0000 7fe6a103e080 20 remove_watcher() i=2 2020-11-29T15:56:23.419 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.420+0000 7fe6a103e080 20 remove_watcher() i=3 2020-11-29T15:56:23.422 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.424+0000 7fe6a103e080 20 remove_watcher() i=4 2020-11-29T15:56:23.425 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.428+0000 7fe6a103e080 20 remove_watcher() i=5 2020-11-29T15:56:23.428 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.432+0000 7fe6a103e080 20 remove_watcher() i=6 2020-11-29T15:56:23.430 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.432+0000 7fe6a103e080 20 remove_watcher() i=7 2020-11-29T15:56:23.432 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.436+0000 7fe65d7f2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-29T15:56:23.449 DEBUG:tasks.util.rgw: json result: [] 2020-11-29T15:56:23.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-29T15:56:23.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-29T15:56:23.450 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-29T15:56:23.533 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:23.533 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:23.543 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.544+0000 7f50abcd6080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-29T15:56:23.543 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.544+0000 7f50abcd6080 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-29T15:56:23.551 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.552+0000 7f4fcb7e6700 20 reqs_thread_entry: start 2020-11-29T15:56:23.551 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.552+0000 7f50abcd6080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.552 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.556+0000 7f50abcd6080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:23.552 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.556+0000 7f50abcd6080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.554 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.556+0000 7f50abcd6080 20 rados_obj.operate() r=0 bl.length=873 2020-11-29T15:56:23.554 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.556+0000 7f50abcd6080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.554 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.556+0000 7f50abcd6080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.555 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.556+0000 7f50abcd6080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.555 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.556+0000 7f50abcd6080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:23.556 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.556+0000 7f50abcd6080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.557 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.560+0000 7f50abcd6080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:23.564 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:56:23.564 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a76bf446-de9f-465c-9e33-558473330d4b", 2020-11-29T15:56:23.564 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default", 2020-11-29T15:56:23.565 INFO:teuthology.orchestra.run.smithi168.stdout: "domain_root": "default.rgw.meta:root", 2020-11-29T15:56:23.565 INFO:teuthology.orchestra.run.smithi168.stdout: "control_pool": "default.rgw.control", 2020-11-29T15:56:23.565 INFO:teuthology.orchestra.run.smithi168.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-29T15:56:23.566 INFO:teuthology.orchestra.run.smithi168.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-29T15:56:23.566 INFO:teuthology.orchestra.run.smithi168.stdout: "log_pool": "default.rgw.log", 2020-11-29T15:56:23.566 INFO:teuthology.orchestra.run.smithi168.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-29T15:56:23.567 INFO:teuthology.orchestra.run.smithi168.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-29T15:56:23.567 INFO:teuthology.orchestra.run.smithi168.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-29T15:56:23.567 INFO:teuthology.orchestra.run.smithi168.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-29T15:56:23.567 INFO:teuthology.orchestra.run.smithi168.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-29T15:56:23.567 INFO:teuthology.orchestra.run.smithi168.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-29T15:56:23.568 INFO:teuthology.orchestra.run.smithi168.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-29T15:56:23.568 INFO:teuthology.orchestra.run.smithi168.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-29T15:56:23.568 INFO:teuthology.orchestra.run.smithi168.stdout: "otp_pool": "default.rgw.otp", 2020-11-29T15:56:23.568 INFO:teuthology.orchestra.run.smithi168.stdout: "system_key": { 2020-11-29T15:56:23.568 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "", 2020-11-29T15:56:23.569 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "" 2020-11-29T15:56:23.569 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:23.569 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_pools": [ 2020-11-29T15:56:23.569 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:23.569 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "default-placement", 2020-11-29T15:56:23.570 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:56:23.570 INFO:teuthology.orchestra.run.smithi168.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-29T15:56:23.570 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": { 2020-11-29T15:56:23.570 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD": { 2020-11-29T15:56:23.570 INFO:teuthology.orchestra.run.smithi168.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-29T15:56:23.571 INFO:teuthology.orchestra.run.smithi168.stdout: "compression_type": "random" 2020-11-29T15:56:23.571 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:23.571 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:23.571 INFO:teuthology.orchestra.run.smithi168.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-29T15:56:23.571 INFO:teuthology.orchestra.run.smithi168.stdout: "index_type": 0 2020-11-29T15:56:23.572 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:23.572 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:23.572 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:23.572 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "", 2020-11-29T15:56:23.572 INFO:teuthology.orchestra.run.smithi168.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-29T15:56:23.573 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:56:23.574 DEBUG:tasks.util.rgw: json result: {'id': 'a76bf446-de9f-465c-9e33-558473330d4b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-29T15:56:23.574 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-11-29T15:56:23.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-11-29T15:56:23.575 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-11-29T15:56:23.649 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:23.649 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:23.661 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.664+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.663 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.664+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.664 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.664+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.665 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.668+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.665 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.668+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.666 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.668+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.666 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.668+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.667 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.668+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:23.693 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 RGWRados::pool_iterate: got zone_info.a76bf446-de9f-465c-9e33-558473330d4b 2020-11-29T15:56:23.693 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 RGWRados::pool_iterate: got zonegroup_info.b74a3860-754a-44af-9945-c221c8c010d8 2020-11-29T15:56:23.693 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 RGWRados::pool_iterate: got zone_names.default 2020-11-29T15:56:23.694 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-29T15:56:23.694 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.694 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:23.695 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.695 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=889 2020-11-29T15:56:23.696 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.696+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.696 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:23.697 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.697 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:23.697 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.698 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.698 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 10 cannot find current period zonegroup using local zonegroup 2020-11-29T15:56:23.699 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.700 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.700 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.700+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.700 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:23.700 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.701 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:23.701 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 zonegroup default 2020-11-29T15:56:23.702 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.702 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.702 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 10 Cannot find current period zone using local zone 2020-11-29T15:56:23.702 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.703 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.703 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.703 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:23.704 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.704 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados_obj.operate() r=0 bl.length=889 2020-11-29T15:56:23.705 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 zone default found 2020-11-29T15:56:23.705 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.704+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.705 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.708+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.705 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.708+0000 7f8343dc4080 20 rados->read ofs=0 len=0 2020-11-29T15:56:23.706 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.708+0000 7f8343dc4080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:23.706 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.708+0000 7f8343dc4080 20 started sync module instance, tier type = 2020-11-29T15:56:23.706 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.708+0000 7f8343dc4080 20 started zone id=a76bf446-de9f-465c-9e33-558473330d4b (name=default) with tier type = 2020-11-29T15:56:23.733 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 20 add_watcher() i=0 2020-11-29T15:56:23.733 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 20 add_watcher() i=1 2020-11-29T15:56:23.733 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 20 add_watcher() i=2 2020-11-29T15:56:23.733 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 20 add_watcher() i=3 2020-11-29T15:56:23.734 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 20 add_watcher() i=4 2020-11-29T15:56:23.734 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 20 add_watcher() i=5 2020-11-29T15:56:23.734 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 20 add_watcher() i=6 2020-11-29T15:56:23.735 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 20 add_watcher() i=7 2020-11-29T15:56:23.735 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.736+0000 7f8343dc4080 2 all 8 watchers are set, enabling cache 2020-11-29T15:56:23.737 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.740+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:23.738 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.740+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-29T15:56:23.738 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.740+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-29T15:56:23.739 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.740+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-29T15:56:23.739 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.740+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-29T15:56:23.739 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.740+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-29T15:56:23.740 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.740+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:23.740 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-29T15:56:23.740 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-29T15:56:23.741 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-29T15:56:23.742 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-29T15:56:23.742 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-29T15:56:23.742 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:23.743 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-29T15:56:23.743 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-29T15:56:23.743 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-29T15:56:23.743 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.744+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-29T15:56:23.744 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.748+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-29T15:56:23.744 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.748+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:23.745 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.748+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-29T15:56:23.745 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.748+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-29T15:56:23.745 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.748+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-29T15:56:23.746 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.748+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-29T15:56:23.746 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.748+0000 7f8343dc4080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-29T15:56:23.746 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.748+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:23.748 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.752+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:23.749 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.752+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:23.752 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.752+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:23.752 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.756+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:23.755 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.756+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:23.756 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.760+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:23.760 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.764+0000 7f8343dc4080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:23.762 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.764+0000 7f8300fe9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-29T15:56:23.762 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.764+0000 7f8343dc4080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-29T15:56:23.762 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.764+0000 7f8343dc4080 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-29T15:56:23.859 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.860+0000 7f825afe5700 20 reqs_thread_entry: start 2020-11-29T15:56:23.860 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.860+0000 7f8258fe1700 20 reqs_thread_entry: start 2020-11-29T15:56:23.860 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.860+0000 7f8343dc4080 20 init_complete bucket index max shards: 11 2020-11-29T15:56:23.861 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.864+0000 7f8343dc4080 10 Started notification manager with: 1 workers 2020-11-29T15:56:23.861 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.864+0000 7f8252ffd700 20 INFO: next queues processing will happen at: Sun Nov 29 15:56:54 2020 2020-11-29T15:56:23.861 INFO:teuthology.orchestra.run.smithi168.stderr: 2020-11-29T15:56:23.861 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.864+0000 7f82527fc700 20 reqs_thread_entry: start 2020-11-29T15:56:23.892 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T15:56:23.892 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.892+0000 7f8343dc4080 20 remove_watcher() i=0 2020-11-29T15:56:23.893 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.892+0000 7f8343dc4080 2 removed watcher, disabling cache 2020-11-29T15:56:23.895 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.896+0000 7f8343dc4080 20 remove_watcher() i=1 2020-11-29T15:56:23.898 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.900+0000 7f8343dc4080 20 remove_watcher() i=2 2020-11-29T15:56:23.902 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.904+0000 7f8343dc4080 20 remove_watcher() i=3 2020-11-29T15:56:23.905 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.908+0000 7f8343dc4080 20 remove_watcher() i=4 2020-11-29T15:56:23.908 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.912+0000 7f8343dc4080 20 remove_watcher() i=5 2020-11-29T15:56:23.910 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.912+0000 7f8343dc4080 20 remove_watcher() i=6 2020-11-29T15:56:23.913 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.916+0000 7f8343dc4080 20 remove_watcher() i=7 2020-11-29T15:56:23.914 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:23.916+0000 7f8300fe9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-29T15:56:23.922 DEBUG:tasks.util.rgw: json result: [] 2020-11-29T15:56:23.922 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-29T15:56:23.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-29T15:56:23.924 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-29T15:56:24.003 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:24.003 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:24.014 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.016+0000 7f86c6f61080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-29T15:56:24.015 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.016+0000 7f86c6f61080 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-29T15:56:24.022 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.024+0000 7f85e7fdf700 20 reqs_thread_entry: start 2020-11-29T15:56:24.023 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.024+0000 7f86c6f61080 20 rados->read ofs=0 len=0 2020-11-29T15:56:24.024 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.028+0000 7f86c6f61080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:24.025 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.028+0000 7f86c6f61080 20 rados->read ofs=0 len=0 2020-11-29T15:56:24.026 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.028+0000 7f86c6f61080 20 rados_obj.operate() r=0 bl.length=889 2020-11-29T15:56:24.026 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.028+0000 7f86c6f61080 20 rados->read ofs=0 len=0 2020-11-29T15:56:24.026 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.028+0000 7f86c6f61080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:24.027 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.028+0000 7f86c6f61080 20 rados->read ofs=0 len=0 2020-11-29T15:56:24.028 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.028+0000 7f86c6f61080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:24.028 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.028+0000 7f86c6f61080 20 rados->read ofs=0 len=0 2020-11-29T15:56:24.029 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:24.032+0000 7f86c6f61080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:24.037 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:56:24.037 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a76bf446-de9f-465c-9e33-558473330d4b", 2020-11-29T15:56:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default", 2020-11-29T15:56:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "domain_root": "default.rgw.meta:root", 2020-11-29T15:56:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "control_pool": "default.rgw.control", 2020-11-29T15:56:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-29T15:56:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-29T15:56:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: "log_pool": "default.rgw.log", 2020-11-29T15:56:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-29T15:56:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-29T15:56:24.040 INFO:teuthology.orchestra.run.smithi168.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-29T15:56:24.040 INFO:teuthology.orchestra.run.smithi168.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-29T15:56:24.041 INFO:teuthology.orchestra.run.smithi168.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-29T15:56:24.041 INFO:teuthology.orchestra.run.smithi168.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-29T15:56:24.042 INFO:teuthology.orchestra.run.smithi168.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-29T15:56:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-29T15:56:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "otp_pool": "default.rgw.otp", 2020-11-29T15:56:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "system_key": { 2020-11-29T15:56:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "", 2020-11-29T15:56:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "" 2020-11-29T15:56:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_pools": [ 2020-11-29T15:56:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "default-placement", 2020-11-29T15:56:24.045 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:56:24.045 INFO:teuthology.orchestra.run.smithi168.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-29T15:56:24.045 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": { 2020-11-29T15:56:24.045 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD": { 2020-11-29T15:56:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-29T15:56:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: "compression_type": "random" 2020-11-29T15:56:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-29T15:56:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: "index_type": 0 2020-11-29T15:56:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:24.048 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:24.048 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "", 2020-11-29T15:56:24.048 INFO:teuthology.orchestra.run.smithi168.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-29T15:56:24.048 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:56:24.049 DEBUG:tasks.util.rgw: json result: {'id': 'a76bf446-de9f-465c-9e33-558473330d4b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-29T15:56:24.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-11-29T15:56:24.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-11-29T15:56:24.051 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-11-29T15:56:24.086 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:24.087 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:24.098 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:24.100+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.779 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.784+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.779 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.784+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.780 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.784+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.780 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.784+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.780 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.784+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.780 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.784+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.791 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.796+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.810 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.812 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.812 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.813 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.813 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 10 cannot find current period zonegroup using local zonegroup 2020-11-29T15:56:25.814 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.814 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.814 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.814 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.814 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 10 Creating default zonegroup 2020-11-29T15:56:25.815 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 20 get_system_obj_state: rctx=0x7fff2a297e90 obj=:.pools.avail state=0x561784c8aa30 s->prefetch_data=0 2020-11-29T15:56:25.815 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.816+0000 7fa8de5a3080 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-11-29T15:56:25.828 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.832+0000 7fa8de5a3080 10 WARNING: store->list_zones() returned r=-2 2020-11-29T15:56:25.828 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.832+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.829 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.832+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.834 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.840+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.835 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.840+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.835 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.840+0000 7fa8de5a3080 10 could not read realm id: (2) No such file or directory 2020-11-29T15:56:25.835 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.840+0000 7fa8de5a3080 10 WARNING: failed to set zone as default, r=-22 2020-11-29T15:56:25.836 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.840+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.836 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.840+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.841 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.844+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.842 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados_obj.operate() r=0 bl.length=873 2020-11-29T15:56:25.843 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.843 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:25.843 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 zonegroup default 2020-11-29T15:56:25.844 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.844 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:25.845 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 10 Cannot find current period zone using local zone 2020-11-29T15:56:25.845 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 10 Using default name default 2020-11-29T15:56:25.845 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.845 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:25.846 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:25.846 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados_obj.operate() r=0 bl.length=873 2020-11-29T15:56:25.847 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 zone default found 2020-11-29T15:56:25.847 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:25.848+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:27.809 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:27.812+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:27.810 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:27.812+0000 7fa8de5a3080 20 rados->read ofs=0 len=0 2020-11-29T15:56:27.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:27.816+0000 7fa8de5a3080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:27.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:27.816+0000 7fa8de5a3080 20 started sync module instance, tier type = 2020-11-29T15:56:27.812 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:27.816+0000 7fa8de5a3080 20 started zone id=0fd54e07-b964-4774-bb91-d1b41f3a40b3 (name=default) with tier type = 2020-11-29T15:56:29.866 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.868+0000 7fa8de5a3080 20 add_watcher() i=0 2020-11-29T15:56:29.867 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.868+0000 7fa8de5a3080 20 add_watcher() i=1 2020-11-29T15:56:29.867 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.868+0000 7fa8de5a3080 20 add_watcher() i=2 2020-11-29T15:56:29.867 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.868+0000 7fa8de5a3080 20 add_watcher() i=3 2020-11-29T15:56:29.867 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.868+0000 7fa8de5a3080 20 add_watcher() i=4 2020-11-29T15:56:29.867 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.868+0000 7fa8de5a3080 20 add_watcher() i=5 2020-11-29T15:56:29.868 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.872+0000 7fa8de5a3080 20 add_watcher() i=6 2020-11-29T15:56:29.868 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.872+0000 7fa8de5a3080 20 add_watcher() i=7 2020-11-29T15:56:29.869 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.872+0000 7fa8de5a3080 2 all 8 watchers are set, enabling cache 2020-11-29T15:56:29.870 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.876+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:29.871 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.876+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:29.871 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.876+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:29.872 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.876+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:29.872 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.876+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:29.875 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.880+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:29.879 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.884+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:29.881 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.884+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:29.886 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.888+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:29.889 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.892+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:29.890 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.896+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:29.894 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.896+0000 7fa8de5a3080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:29.896 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.900+0000 7fa89bfdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-29T15:56:29.896 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.900+0000 7fa8de5a3080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-29T15:56:29.896 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:29.900+0000 7fa8de5a3080 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-29T15:56:33.843 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:33.843+0000 7fa8de5a3080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-11-29T15:56:33.989 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:33.991+0000 7fa7f27e4700 20 reqs_thread_entry: start 2020-11-29T15:56:33.989 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:33.991+0000 7fa8de5a3080 20 init_complete bucket index max shards: 11 2020-11-29T15:56:33.990 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:33.991+0000 7fa7e3fff700 20 reqs_thread_entry: start 2020-11-29T15:56:33.990 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:33.991+0000 7fa8de5a3080 10 Started notification manager with: 1 workers 2020-11-29T15:56:33.990 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:33.991+0000 7fa7e27fc700 20 INFO: next queues processing will happen at: Sun Nov 29 15:57:04 2020 2020-11-29T15:56:33.990 INFO:teuthology.orchestra.run.smithi184.stderr: 2020-11-29T15:56:33.990 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:33.991+0000 7fa7e1ffb700 20 reqs_thread_entry: start 2020-11-29T15:56:34.021 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-11-29T15:56:34.022 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.023+0000 7fa8de5a3080 20 remove_watcher() i=0 2020-11-29T15:56:34.022 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.023+0000 7fa8de5a3080 2 removed watcher, disabling cache 2020-11-29T15:56:34.024 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.027+0000 7fa8de5a3080 20 remove_watcher() i=1 2020-11-29T15:56:34.027 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.031+0000 7fa8de5a3080 20 remove_watcher() i=2 2020-11-29T15:56:34.030 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.031+0000 7fa8de5a3080 20 remove_watcher() i=3 2020-11-29T15:56:34.032 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.035+0000 7fa8de5a3080 20 remove_watcher() i=4 2020-11-29T15:56:34.035 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.039+0000 7fa8de5a3080 20 remove_watcher() i=5 2020-11-29T15:56:34.038 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.039+0000 7fa8de5a3080 20 remove_watcher() i=6 2020-11-29T15:56:34.040 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.043+0000 7fa8de5a3080 20 remove_watcher() i=7 2020-11-29T15:56:34.041 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.043+0000 7fa89bfdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-29T15:56:34.055 DEBUG:tasks.util.rgw: json result: [] 2020-11-29T15:56:34.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-29T15:56:34.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-29T15:56:34.056 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-29T15:56:34.138 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:34.139 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:34.147 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.151+0000 7f76518ac080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-29T15:56:34.147 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.151+0000 7f76518ac080 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-29T15:56:34.155 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.159+0000 7f7571fe3700 20 reqs_thread_entry: start 2020-11-29T15:56:34.156 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.159+0000 7f76518ac080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.158 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.159+0000 7f76518ac080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.158 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.159+0000 7f76518ac080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.159 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.163+0000 7f76518ac080 20 rados_obj.operate() r=0 bl.length=873 2020-11-29T15:56:34.159 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.163+0000 7f76518ac080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.160 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.163+0000 7f76518ac080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.160 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.163+0000 7f76518ac080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.161 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.163+0000 7f76518ac080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.161 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.163+0000 7f76518ac080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.163 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.167+0000 7f76518ac080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:34.171 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:56:34.171 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "0fd54e07-b964-4774-bb91-d1b41f3a40b3", 2020-11-29T15:56:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default", 2020-11-29T15:56:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "default.rgw.meta:root", 2020-11-29T15:56:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "default.rgw.control", 2020-11-29T15:56:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-29T15:56:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-29T15:56:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "default.rgw.log", 2020-11-29T15:56:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-29T15:56:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-29T15:56:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-29T15:56:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-29T15:56:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-29T15:56:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-29T15:56:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-29T15:56:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-29T15:56:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "default.rgw.otp", 2020-11-29T15:56:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T15:56:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "", 2020-11-29T15:56:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "" 2020-11-29T15:56:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:56:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T15:56:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:56:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T15:56:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:56:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-29T15:56:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T15:56:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T15:56:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-29T15:56:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T15:56:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:56:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-29T15:56:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T15:56:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:56:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2020-11-29T15:56:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-29T15:56:34.179 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:56:34.180 DEBUG:tasks.util.rgw: json result: {'id': '0fd54e07-b964-4774-bb91-d1b41f3a40b3', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-29T15:56:34.181 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-11-29T15:56:34.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-11-29T15:56:34.181 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-11-29T15:56:34.249 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:34.249 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:34.259 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.263+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.260 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.263+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.261 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.263+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.261 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.263+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.262 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.263+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.262 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.263+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.262 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.263+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.262 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.263+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.278 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.279+0000 7fd81a075080 20 RGWRados::pool_iterate: got zone_info.0fd54e07-b964-4774-bb91-d1b41f3a40b3 2020-11-29T15:56:34.279 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.279+0000 7fd81a075080 20 RGWRados::pool_iterate: got zonegroup_info.b8e01262-a7ba-4e0c-aebf-c2dda746b1ec 2020-11-29T15:56:34.279 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.279+0000 7fd81a075080 20 RGWRados::pool_iterate: got zone_names.default 2020-11-29T15:56:34.279 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.279+0000 7fd81a075080 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-11-29T15:56:34.279 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.279+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.280 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.283+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.280 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.283+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.283 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=889 2020-11-29T15:56:34.283 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.283 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.283 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.284 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:34.284 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.285 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.285 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 10 cannot find current period zonegroup using local zonegroup 2020-11-29T15:56:34.285 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.286 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.286 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.286 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.287 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.287 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:34.288 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 zonegroup default 2020-11-29T15:56:34.288 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.287+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.288 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.288 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 10 Cannot find current period zone using local zone 2020-11-29T15:56:34.288 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.289 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.289 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.290 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.290 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.290 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados_obj.operate() r=0 bl.length=889 2020-11-29T15:56:34.291 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 zone default found 2020-11-29T15:56:34.291 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.291 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.291 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.291 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.292 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 started sync module instance, tier type = 2020-11-29T15:56:34.292 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.291+0000 7fd81a075080 20 started zone id=0fd54e07-b964-4774-bb91-d1b41f3a40b3 (name=default) with tier type = 2020-11-29T15:56:34.322 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.323+0000 7fd81a075080 20 add_watcher() i=0 2020-11-29T15:56:34.323 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.323+0000 7fd81a075080 20 add_watcher() i=1 2020-11-29T15:56:34.323 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.323+0000 7fd81a075080 20 add_watcher() i=2 2020-11-29T15:56:34.323 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.323+0000 7fd81a075080 20 add_watcher() i=3 2020-11-29T15:56:34.323 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.323+0000 7fd81a075080 20 add_watcher() i=4 2020-11-29T15:56:34.323 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.323+0000 7fd81a075080 20 add_watcher() i=5 2020-11-29T15:56:34.324 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.327+0000 7fd81a075080 20 add_watcher() i=6 2020-11-29T15:56:34.325 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.327+0000 7fd81a075080 20 add_watcher() i=7 2020-11-29T15:56:34.325 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.327+0000 7fd81a075080 2 all 8 watchers are set, enabling cache 2020-11-29T15:56:34.327 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:34.328 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-29T15:56:34.328 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-29T15:56:34.329 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-29T15:56:34.329 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-29T15:56:34.330 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-29T15:56:34.330 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:34.330 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-29T15:56:34.331 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.331+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-29T15:56:34.331 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-29T15:56:34.331 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-29T15:56:34.332 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-29T15:56:34.332 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:34.333 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-29T15:56:34.333 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-29T15:56:34.333 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-29T15:56:34.334 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-29T15:56:34.334 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-29T15:56:34.334 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:34.335 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1165 entering: tid=1 2020-11-29T15:56:34.335 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.335+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1187 max_entries=1 tid=1 2020-11-29T15:56:34.335 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.339+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1199 missing part, rereading metadata tid= 1 2020-11-29T15:56:34.336 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.339+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::read_meta(uint64_t, optional_yield):934 entering: tid=1 2020-11-29T15:56:34.336 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.339+0000 7fd81a075080 20 int rgw::cls::fifo::FIFO::list(int, std::optional >, std::vector*, bool*, optional_yield):1221 assuming part was not written yet, so end of data: tid=1 2020-11-29T15:56:34.336 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.339+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:34.338 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.339+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:34.338 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.343+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:34.343 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.347+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:34.344 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.347+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:34.346 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.347+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:34.347 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.351+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):918 entering 2020-11-29T15:56:34.349 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.351+0000 7fd81a075080 20 static int rgw::cls::fifo::FIFO::open(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):875 entering 2020-11-29T15:56:34.350 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.351+0000 7fd7d77de700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-29T15:56:34.350 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.351+0000 7fd81a075080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-29T15:56:34.350 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.351+0000 7fd81a075080 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-29T15:56:34.446 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.447+0000 7fd731fe3700 20 reqs_thread_entry: start 2020-11-29T15:56:34.447 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.447+0000 7fd81a075080 20 init_complete bucket index max shards: 11 2020-11-29T15:56:34.447 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.447+0000 7fd72b7fe700 20 reqs_thread_entry: start 2020-11-29T15:56:34.448 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.451+0000 7fd81a075080 10 Started notification manager with: 1 workers 2020-11-29T15:56:34.448 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.451+0000 7fd729ffb700 20 INFO: next queues processing will happen at: Sun Nov 29 15:57:04 2020 2020-11-29T15:56:34.448 INFO:teuthology.orchestra.run.smithi184.stderr: 2020-11-29T15:56:34.448 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.451+0000 7fd7297fa700 20 reqs_thread_entry: start 2020-11-29T15:56:34.473 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-11-29T15:56:34.473 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.475+0000 7fd81a075080 20 remove_watcher() i=0 2020-11-29T15:56:34.474 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.475+0000 7fd81a075080 2 removed watcher, disabling cache 2020-11-29T15:56:34.475 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.479+0000 7fd81a075080 20 remove_watcher() i=1 2020-11-29T15:56:34.478 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.479+0000 7fd81a075080 20 remove_watcher() i=2 2020-11-29T15:56:34.480 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.483+0000 7fd81a075080 20 remove_watcher() i=3 2020-11-29T15:56:34.483 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.487+0000 7fd81a075080 20 remove_watcher() i=4 2020-11-29T15:56:34.485 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.487+0000 7fd81a075080 20 remove_watcher() i=5 2020-11-29T15:56:34.488 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.491+0000 7fd81a075080 20 remove_watcher() i=6 2020-11-29T15:56:34.490 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.495+0000 7fd81a075080 20 remove_watcher() i=7 2020-11-29T15:56:34.492 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.495+0000 7fd7d77de700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-11-29T15:56:34.499 DEBUG:tasks.util.rgw: json result: [] 2020-11-29T15:56:34.499 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-29T15:56:34.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-11-29T15:56:34.500 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-11-29T15:56:34.582 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:34.582 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:34.595 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.595+0000 7f85d9100080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-11-29T15:56:34.595 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.595+0000 7f85d9100080 20 check_secure_mon_conn(): mode 1 is insecure 2020-11-29T15:56:34.601 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.603+0000 7f84f8fe1700 20 reqs_thread_entry: start 2020-11-29T15:56:34.601 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.603+0000 7f85d9100080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.602 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.603+0000 7f85d9100080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.603 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.603+0000 7f85d9100080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.604 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.607+0000 7f85d9100080 20 rados_obj.operate() r=0 bl.length=889 2020-11-29T15:56:34.605 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.607+0000 7f85d9100080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.605 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.607+0000 7f85d9100080 20 rados_obj.operate() r=-2 bl.length=0 2020-11-29T15:56:34.605 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.607+0000 7f85d9100080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.606 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.607+0000 7f85d9100080 20 rados_obj.operate() r=0 bl.length=46 2020-11-29T15:56:34.606 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.607+0000 7f85d9100080 20 rados->read ofs=0 len=0 2020-11-29T15:56:34.608 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:34.611+0000 7f85d9100080 20 rados_obj.operate() r=0 bl.length=358 2020-11-29T15:56:34.616 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:56:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "0fd54e07-b964-4774-bb91-d1b41f3a40b3", 2020-11-29T15:56:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default", 2020-11-29T15:56:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "default.rgw.meta:root", 2020-11-29T15:56:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "default.rgw.control", 2020-11-29T15:56:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "default.rgw.log:gc", 2020-11-29T15:56:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "default.rgw.log:lc", 2020-11-29T15:56:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "default.rgw.log", 2020-11-29T15:56:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-11-29T15:56:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-11-29T15:56:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "default.rgw.meta:roles", 2020-11-29T15:56:34.618 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-11-29T15:56:34.619 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-11-29T15:56:34.619 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-11-29T15:56:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-11-29T15:56:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-11-29T15:56:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "default.rgw.otp", 2020-11-29T15:56:34.621 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T15:56:34.621 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "", 2020-11-29T15:56:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "" 2020-11-29T15:56:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:56:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T15:56:34.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:56:34.623 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T15:56:34.623 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:56:34.623 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "default.rgw.buckets.index", 2020-11-29T15:56:34.624 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T15:56:34.624 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T15:56:34.624 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "default.rgw.buckets.data", 2020-11-29T15:56:34.624 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T15:56:34.624 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:34.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:56:34.625 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-11-29T15:56:34.626 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T15:56:34.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:34.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:34.627 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:56:34.627 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2020-11-29T15:56:34.628 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "default.rgw.log:notif" 2020-11-29T15:56:34.628 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:56:34.629 DEBUG:tasks.util.rgw: json result: {'id': '0fd54e07-b964-4774-bb91-d1b41f3a40b3', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-11-29T15:56:34.630 INFO:tasks.rgw:Starting rgw... 2020-11-29T15:56:34.630 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-11-29T15:56:34.631 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-29T15:56:34.631 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-29T15:56:34.632 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-29T15:56:34.635 INFO:tasks.rgw.c1.client.0:Started 2020-11-29T15:56:34.635 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-11-29T15:56:34.635 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-29T15:56:34.636 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-29T15:56:34.636 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-29T15:56:34.638 INFO:tasks.rgw.c1.client.1:Started 2020-11-29T15:56:34.639 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-11-29T15:56:34.639 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-29T15:56:34.639 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-29T15:56:34.640 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-29T15:56:34.674 INFO:tasks.rgw.c2.client.0:Started 2020-11-29T15:56:34.674 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-11-29T15:56:34.674 INFO:tasks.rgw:Using beast as radosgw frontend 2020-11-29T15:56:34.675 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-29T15:56:34.675 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-29T15:56:34.677 INFO:tasks.rgw.c2.client.1:Started 2020-11-29T15:56:34.678 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi168.front.sepia.ceph.com:8000/ 2020-11-29T15:56:34.678 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000/ 2020-11-29T15:56:34.696 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:56:34.697 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:56:34.700 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi168.front.sepia.ceph.com port 8000: Connection refused 2020-11-29T15:56:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-29T15:56:35.702 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000/ 2020-11-29T15:56:35.717 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:56:35.717 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:56:35.723 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-29T15:56:35.724 INFO:teuthology.orchestra.run.smithi168.stdout:anonymous 2020-11-29T15:56:35.724 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi168.front.sepia.ceph.com:8001/ 2020-11-29T15:56:35.725 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001/ 2020-11-29T15:56:35.779 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:56:35.780 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:56:35.791 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-29T15:56:35.791 INFO:teuthology.orchestra.run.smithi168.stdout:anonymous 2020-11-29T15:56:35.791 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi184.front.sepia.ceph.com:8000/ 2020-11-29T15:56:35.792 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000/ 2020-11-29T15:56:35.809 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:56:35.809 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:56:35.815 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-29T15:56:35.816 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2020-11-29T15:56:35.816 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi184.front.sepia.ceph.com:8001/ 2020-11-29T15:56:35.816 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8001/ 2020-11-29T15:56:35.873 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:56:35.873 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:56:35.879 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-29T15:56:35.880 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2020-11-29T15:56:35.881 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-11-29T15:56:36.279 INFO:tasks.rgw_multisite:creating realm.. 2020-11-29T15:56:36.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:36.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:36.282 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:36.315 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:36.315 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:36.371 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:56:36.371 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:56:36.372 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-realm", 2020-11-29T15:56:36.372 INFO:teuthology.orchestra.run.smithi168.stdout: "current_period": "f85496da-377c-48c0-b59c-86bb7b911fb2", 2020-11-29T15:56:36.372 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 1 2020-11-29T15:56:36.372 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:56:36.374 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-11-29T15:56:36.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:36.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:36.375 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:36.453 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:36.454 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:36.478 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:56:36.477+0000 7f693b311080 -1 old format 2020-11-29T15:56:36.493 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:56:36.493 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "test-zonegroup", 2020-11-29T15:56:36.493 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zonegroup", 2020-11-29T15:56:36.493 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "", 2020-11-29T15:56:36.494 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "true", 2020-11-29T15:56:36.494 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T15:56:36.494 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:56:36.494 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:36.495 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T15:56:36.495 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T15:56:36.495 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "", 2020-11-29T15:56:36.495 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [], 2020-11-29T15:56:36.496 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [], 2020-11-29T15:56:36.496 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "", 2020-11-29T15:56:36.496 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:56:36.496 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T15:56:36.496 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T15:56:36.497 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:36.497 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:56:36.498 INFO:tasks.rgw_multisite:creating master zone.. 2020-11-29T15:56:36.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--endpoints', 'http://smithi168.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:36.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--endpoints', 'http://smithi168.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:36.499 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --endpoints http://smithi168.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:36.583 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:36.583 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:36.667 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:56:36.668 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:56:36.668 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T15:56:36.668 INFO:teuthology.orchestra.run.smithi168.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-29T15:56:36.668 INFO:teuthology.orchestra.run.smithi168.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-29T15:56:36.669 INFO:teuthology.orchestra.run.smithi168.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-29T15:56:36.669 INFO:teuthology.orchestra.run.smithi168.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-29T15:56:36.669 INFO:teuthology.orchestra.run.smithi168.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-29T15:56:36.669 INFO:teuthology.orchestra.run.smithi168.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-29T15:56:36.669 INFO:teuthology.orchestra.run.smithi168.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-29T15:56:36.670 INFO:teuthology.orchestra.run.smithi168.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-29T15:56:36.670 INFO:teuthology.orchestra.run.smithi168.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-29T15:56:36.670 INFO:teuthology.orchestra.run.smithi168.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-29T15:56:36.671 INFO:teuthology.orchestra.run.smithi168.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-29T15:56:36.671 INFO:teuthology.orchestra.run.smithi168.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-29T15:56:36.671 INFO:teuthology.orchestra.run.smithi168.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-29T15:56:36.671 INFO:teuthology.orchestra.run.smithi168.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-29T15:56:36.671 INFO:teuthology.orchestra.run.smithi168.stdout: "system_key": { 2020-11-29T15:56:36.672 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:56:36.672 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:56:36.672 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:36.673 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_pools": [ 2020-11-29T15:56:36.673 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:36.673 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "default-placement", 2020-11-29T15:56:36.673 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:56:36.674 INFO:teuthology.orchestra.run.smithi168.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-29T15:56:36.674 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": { 2020-11-29T15:56:36.674 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD": { 2020-11-29T15:56:36.675 INFO:teuthology.orchestra.run.smithi168.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-11-29T15:56:36.675 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:36.675 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:36.675 INFO:teuthology.orchestra.run.smithi168.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-29T15:56:36.675 INFO:teuthology.orchestra.run.smithi168.stdout: "index_type": 0 2020-11-29T15:56:36.676 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:36.676 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:36.676 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:36.676 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:56:36.677 INFO:teuthology.orchestra.run.smithi168.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-29T15:56:36.677 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:56:36.678 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-11-29T15:56:37.226 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-11-29T15:56:37.244 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-11-29T15:56:38.245 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-11-29T15:56:38.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:38.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:38.280 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:38.320 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:38.321 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:38.350 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:56:38.350 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:56:38.351 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T15:56:38.351 INFO:teuthology.orchestra.run.smithi168.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-29T15:56:38.351 INFO:teuthology.orchestra.run.smithi168.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-29T15:56:38.351 INFO:teuthology.orchestra.run.smithi168.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-29T15:56:38.351 INFO:teuthology.orchestra.run.smithi168.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-29T15:56:38.352 INFO:teuthology.orchestra.run.smithi168.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-29T15:56:38.352 INFO:teuthology.orchestra.run.smithi168.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-29T15:56:38.352 INFO:teuthology.orchestra.run.smithi168.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-29T15:56:38.352 INFO:teuthology.orchestra.run.smithi168.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-29T15:56:38.352 INFO:teuthology.orchestra.run.smithi168.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-29T15:56:38.353 INFO:teuthology.orchestra.run.smithi168.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-29T15:56:38.353 INFO:teuthology.orchestra.run.smithi168.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-29T15:56:38.353 INFO:teuthology.orchestra.run.smithi168.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-29T15:56:38.353 INFO:teuthology.orchestra.run.smithi168.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-29T15:56:38.354 INFO:teuthology.orchestra.run.smithi168.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-29T15:56:38.354 INFO:teuthology.orchestra.run.smithi168.stdout: "system_key": { 2020-11-29T15:56:38.354 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:56:38.354 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:56:38.354 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:38.355 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_pools": [ 2020-11-29T15:56:38.355 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:38.355 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "default-placement", 2020-11-29T15:56:38.355 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:56:38.355 INFO:teuthology.orchestra.run.smithi168.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-29T15:56:38.356 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": { 2020-11-29T15:56:38.356 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD": { 2020-11-29T15:56:38.356 INFO:teuthology.orchestra.run.smithi168.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-29T15:56:38.356 INFO:teuthology.orchestra.run.smithi168.stdout: "compression_type": "random" 2020-11-29T15:56:38.357 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:38.357 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:38.357 INFO:teuthology.orchestra.run.smithi168.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-29T15:56:38.357 INFO:teuthology.orchestra.run.smithi168.stdout: "index_type": 0 2020-11-29T15:56:38.357 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:38.358 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:38.358 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:38.358 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "", 2020-11-29T15:56:38.358 INFO:teuthology.orchestra.run.smithi168.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-29T15:56:38.359 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:56:38.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:38.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:38.360 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:38.440 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:38.441 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:46.507 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:56:46.507 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:56:46.507 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 1, 2020-11-29T15:56:46.508 INFO:teuthology.orchestra.run.smithi168.stdout: "predecessor_uuid": "f85496da-377c-48c0-b59c-86bb7b911fb2", 2020-11-29T15:56:46.508 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": [], 2020-11-29T15:56:46.508 INFO:teuthology.orchestra.run.smithi168.stdout: "period_map": { 2020-11-29T15:56:46.508 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:56:46.508 INFO:teuthology.orchestra.run.smithi168.stdout: "zonegroups": [ 2020-11-29T15:56:46.509 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:46.509 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "test-zonegroup", 2020-11-29T15:56:46.509 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zonegroup", 2020-11-29T15:56:46.509 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "", 2020-11-29T15:56:46.510 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "true", 2020-11-29T15:56:46.510 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T15:56:46.510 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:56:46.510 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:46.510 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T15:56:46.511 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T15:56:46.511 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:56:46.511 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [ 2020-11-29T15:56:46.511 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:46.512 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:56:46.512 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T15:56:46.512 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T15:56:46.512 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:56:46.512 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:46.513 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T15:56:46.513 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "false", 2020-11-29T15:56:46.513 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:56:46.513 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T15:56:46.514 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T15:56:46.514 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T15:56:46.514 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T15:56:46.514 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T15:56:46.514 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:46.515 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:46.515 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [ 2020-11-29T15:56:46.515 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:46.515 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default-placement", 2020-11-29T15:56:46.516 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-11-29T15:56:46.516 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": [ 2020-11-29T15:56:46.516 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD" 2020-11-29T15:56:46.516 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:56:46.516 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:46.517 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:46.517 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "default-placement", 2020-11-29T15:56:46.517 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:56:46.517 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T15:56:46.518 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T15:56:46.518 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:46.518 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:46.518 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:46.518 INFO:teuthology.orchestra.run.smithi168.stdout: "short_zone_ids": [ 2020-11-29T15:56:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:56:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 1785095687 2020-11-29T15:56:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:56:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T15:56:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:56:46.521 INFO:teuthology.orchestra.run.smithi168.stdout: "period_config": { 2020-11-29T15:56:46.521 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_quota": { 2020-11-29T15:56:46.521 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T15:56:46.521 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T15:56:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T15:56:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T15:56:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T15:56:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: "user_quota": { 2020-11-29T15:56:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T15:56:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T15:56:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T15:56:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T15:56:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T15:56:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:46.525 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:56:46.525 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_name": "test-realm", 2020-11-29T15:56:46.525 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:56:46.525 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:56:46.526 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-11-29T15:56:46.527 INFO:teuthology.orchestra.run:waiting for 300 2020-11-29T15:56:46.571 INFO:tasks.rgw.c1.client.0.smithi168.stdout:2020-11-29T15:56:46.573+0000 7f8f84f6e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 14789) UID: 0 2020-11-29T15:56:46.571 INFO:tasks.rgw.c1.client.0.smithi168.stdout:2020-11-29T15:56:46.573+0000 7f8f9ca210c0 -1 shutting down 2020-11-29T15:56:51.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-11-29T15:56:52.629 INFO:tasks.rgw.c1.client.0:Stopped 2020-11-29T15:56:52.629 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-11-29T15:56:52.630 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-11-29T15:56:52.632 INFO:tasks.rgw.c1.client.0:Started 2020-11-29T15:56:52.633 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000 2020-11-29T15:56:52.649 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:56:52.649 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:56:52.653 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi168.front.sepia.ceph.com port 8000: Connection refused 2020-11-29T15:56:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-29T15:56:53.655 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8000 2020-11-29T15:56:53.670 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:56:53.670 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:56:53.676 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 35666 2020-11-29T15:56:53.677 INFO:teuthology.orchestra.run.smithi168.stdout:anonymous 2020-11-29T15:56:53.677 INFO:tasks.rgw_multisite:creating admin user.. 2020-11-29T15:56:53.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:53.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:53.679 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:53.768 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:53.768 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:53.981 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:56:53.981 INFO:teuthology.orchestra.run.smithi168.stdout: "user_id": "realm-admin", 2020-11-29T15:56:53.981 INFO:teuthology.orchestra.run.smithi168.stdout: "display_name": "Realm Admin", 2020-11-29T15:56:53.981 INFO:teuthology.orchestra.run.smithi168.stdout: "email": "", 2020-11-29T15:56:53.982 INFO:teuthology.orchestra.run.smithi168.stdout: "suspended": 0, 2020-11-29T15:56:53.982 INFO:teuthology.orchestra.run.smithi168.stdout: "max_buckets": 1000, 2020-11-29T15:56:53.982 INFO:teuthology.orchestra.run.smithi168.stdout: "subusers": [], 2020-11-29T15:56:53.982 INFO:teuthology.orchestra.run.smithi168.stdout: "keys": [ 2020-11-29T15:56:53.983 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:56:53.983 INFO:teuthology.orchestra.run.smithi168.stdout: "user": "realm-admin", 2020-11-29T15:56:53.983 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:56:53.983 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:56:53.984 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:56:53.984 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:56:53.984 INFO:teuthology.orchestra.run.smithi168.stdout: "swift_keys": [], 2020-11-29T15:56:53.984 INFO:teuthology.orchestra.run.smithi168.stdout: "caps": [], 2020-11-29T15:56:53.984 INFO:teuthology.orchestra.run.smithi168.stdout: "op_mask": "read, write, delete", 2020-11-29T15:56:53.985 INFO:teuthology.orchestra.run.smithi168.stdout: "system": "true", 2020-11-29T15:56:53.985 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "", 2020-11-29T15:56:53.985 INFO:teuthology.orchestra.run.smithi168.stdout: "default_storage_class": "", 2020-11-29T15:56:53.985 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_tags": [], 2020-11-29T15:56:53.985 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_quota": { 2020-11-29T15:56:53.986 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T15:56:53.986 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T15:56:53.986 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T15:56:53.986 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T15:56:53.986 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T15:56:53.987 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:53.987 INFO:teuthology.orchestra.run.smithi168.stdout: "user_quota": { 2020-11-29T15:56:53.987 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T15:56:53.987 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T15:56:53.987 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T15:56:53.988 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T15:56:53.989 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T15:56:53.989 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:56:53.989 INFO:teuthology.orchestra.run.smithi168.stdout: "temp_url_keys": [], 2020-11-29T15:56:53.989 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "rgw", 2020-11-29T15:56:53.990 INFO:teuthology.orchestra.run.smithi168.stdout: "mfa_ids": [] 2020-11-29T15:56:53.990 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:56:53.990 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:56:54.009 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-29T15:56:54.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi168.front.sepia.ceph.com:8000', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:54.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi168.front.sepia.ceph.com:8000', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:54.011 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi168.front.sepia.ceph.com:8000 --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:54.055 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:54.055 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:54.169 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:56:54.170 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:56:54.170 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-realm", 2020-11-29T15:56:54.170 INFO:teuthology.orchestra.run.smithi184.stdout: "current_period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:56:54.170 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2 2020-11-29T15:56:54.171 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:56:54.172 INFO:tasks.rgw_multisite:creating zone.. 2020-11-29T15:56:54.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:54.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:54.173 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --endpoints http://smithi184.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:54.258 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:54.259 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:54.321 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:54.320+0000 7f29eed76080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T15:56:54.321 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:56:54.320+0000 7f29eed76080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T15:56:54.332 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:56:54.333 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T15:56:54.333 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T15:56:54.333 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-29T15:56:54.333 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-29T15:56:54.333 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-29T15:56:54.334 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-29T15:56:54.334 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-29T15:56:54.334 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-29T15:56:54.334 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-29T15:56:54.335 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-29T15:56:54.335 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-29T15:56:54.335 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-29T15:56:54.335 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-29T15:56:54.335 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-29T15:56:54.336 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-29T15:56:54.336 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-29T15:56:54.336 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T15:56:54.336 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:56:54.336 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:56:54.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:56:54.337 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T15:56:54.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:56:54.337 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T15:56:54.337 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:56:54.338 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-29T15:56:54.338 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T15:56:54.338 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T15:56:54.338 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-11-29T15:56:54.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:54.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:56:54.339 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-29T15:56:54.339 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T15:56:54.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:54.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:54.340 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:56:54.340 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:56:54.340 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-29T15:56:54.340 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:56:54.341 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-11-29T15:56:54.829 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-11-29T15:56:54.838 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-11-29T15:56:55.818 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-11-29T15:56:55.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:55.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:55.848 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:55.889 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:55.890 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:55.915 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:56:55.916 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T15:56:55.916 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T15:56:55.916 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-29T15:56:55.916 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-29T15:56:55.917 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-29T15:56:55.917 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-29T15:56:55.917 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-29T15:56:55.917 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-29T15:56:55.917 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-29T15:56:55.918 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-29T15:56:55.918 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-29T15:56:55.918 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-29T15:56:55.918 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-29T15:56:55.919 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-29T15:56:55.919 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-29T15:56:55.919 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-29T15:56:55.919 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T15:56:55.919 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:56:55.920 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:56:55.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:56:55.920 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T15:56:55.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:56:55.921 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T15:56:55.921 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:56:55.921 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-29T15:56:55.922 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T15:56:55.922 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T15:56:55.922 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-29T15:56:55.923 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T15:56:55.923 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:55.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:56:55.923 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-29T15:56:55.924 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T15:56:55.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:55.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:56:55.924 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:56:55.924 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2020-11-29T15:56:55.924 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-29T15:56:55.925 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:56:55.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:55.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:56:55.927 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T15:56:56.005 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:56:56.006 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:56:57.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-11-29T15:57:04.012 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T15:57:04.104 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:57:04.105 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:04.105 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2, 2020-11-29T15:57:04.105 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f85496da-377c-48c0-b59c-86bb7b911fb2", 2020-11-29T15:57:04.106 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [], 2020-11-29T15:57:04.106 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-11-29T15:57:04.106 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:04.106 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-11-29T15:57:04.106 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:04.107 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-11-29T15:57:04.107 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-11-29T15:57:04.107 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-11-29T15:57:04.107 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-11-29T15:57:04.108 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T15:57:04.108 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:57:04.108 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:04.108 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-11-29T15:57:04.108 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-11-29T15:57:04.109 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:04.109 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-11-29T15:57:04.109 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:04.109 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T15:57:04.109 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T15:57:04.110 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T15:57:04.110 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T15:57:04.110 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:04.110 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T15:57:04.111 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T15:57:04.111 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:04.111 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T15:57:04.111 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T15:57:04.112 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T15:57:04.112 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T15:57:04.112 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T15:57:04.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:04.112 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:04.113 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:04.113 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-11-29T15:57:04.113 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T15:57:04.113 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:57:04.113 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:04.114 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T15:57:04.114 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T15:57:04.114 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:04.114 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T15:57:04.115 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T15:57:04.115 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T15:57:04.115 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T15:57:04.115 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T15:57:04.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:04.116 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:04.116 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-11-29T15:57:04.116 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:04.116 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-11-29T15:57:04.116 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-11-29T15:57:04.117 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-11-29T15:57:04.117 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-11-29T15:57:04.117 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:57:04.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:04.117 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:04.118 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-11-29T15:57:04.118 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:04.118 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-11-29T15:57:04.118 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-11-29T15:57:04.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:04.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:04.119 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:04.119 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-11-29T15:57:04.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:04.120 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T15:57:04.120 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2491292681 2020-11-29T15:57:04.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:04.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:04.120 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1785095687 2020-11-29T15:57:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:57:04.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:04.122 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T15:57:04.122 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:04.122 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-11-29T15:57:04.122 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-11-29T15:57:04.122 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T15:57:04.123 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T15:57:04.123 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T15:57:04.123 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T15:57:04.123 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T15:57:04.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:04.124 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-11-29T15:57:04.124 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T15:57:04.124 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T15:57:04.124 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T15:57:04.125 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T15:57:04.125 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T15:57:04.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:04.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:04.125 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:04.126 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-11-29T15:57:04.126 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:04.126 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:57:04.127 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-11-29T15:57:04.127 INFO:teuthology.orchestra.run:waiting for 300 2020-11-29T15:57:04.170 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T15:57:04.172+0000 7fdc4b7fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 14800) UID: 0 2020-11-29T15:57:04.170 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T15:57:04.172+0000 7fdc6f2f60c0 -1 shutting down 2020-11-29T15:57:04.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-11-29T15:57:10.229 INFO:tasks.rgw.c2.client.0:Stopped 2020-11-29T15:57:10.229 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-11-29T15:57:10.230 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-11-29T15:57:10.232 INFO:tasks.rgw.c2.client.0:Started 2020-11-29T15:57:10.233 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2020-11-29T15:57:10.251 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:57:10.251 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:57:10.255 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2020-11-29T15:57:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-29T15:57:10.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-11-29T15:57:11.257 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2020-11-29T15:57:11.272 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:57:11.272 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:57:11.279 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-29T15:57:11.280 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2020-11-29T15:57:11.280 INFO:tasks.rgw_multisite:creating zone.. 2020-11-29T15:57:11.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--endpoints', 'http://smithi168.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:11.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--endpoints', 'http://smithi168.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:11.282 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --endpoints http://smithi168.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:57:11.318 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:11.318 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:11.385 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:57:11.385+0000 7f07c9241080 0 failed reading obj info from .rgw.root:zone_info.3c165286-3738-4e0f-b399-acd859302998: (2) No such file or directory 2020-11-29T15:57:11.385 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T15:57:11.385+0000 7f07c9241080 0 WARNING: could not read zone params for zone id=3c165286-3738-4e0f-b399-acd859302998 name=test-zone2 2020-11-29T15:57:11.393 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:57:11.393 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T15:57:11.394 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone3", 2020-11-29T15:57:11.394 INFO:teuthology.orchestra.run.smithi168.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-29T15:57:11.394 INFO:teuthology.orchestra.run.smithi168.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-29T15:57:11.394 INFO:teuthology.orchestra.run.smithi168.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-29T15:57:11.395 INFO:teuthology.orchestra.run.smithi168.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-29T15:57:11.395 INFO:teuthology.orchestra.run.smithi168.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-29T15:57:11.395 INFO:teuthology.orchestra.run.smithi168.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-29T15:57:11.395 INFO:teuthology.orchestra.run.smithi168.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-29T15:57:11.396 INFO:teuthology.orchestra.run.smithi168.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-29T15:57:11.396 INFO:teuthology.orchestra.run.smithi168.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-29T15:57:11.396 INFO:teuthology.orchestra.run.smithi168.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-29T15:57:11.396 INFO:teuthology.orchestra.run.smithi168.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-29T15:57:11.396 INFO:teuthology.orchestra.run.smithi168.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-29T15:57:11.397 INFO:teuthology.orchestra.run.smithi168.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-29T15:57:11.397 INFO:teuthology.orchestra.run.smithi168.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-29T15:57:11.397 INFO:teuthology.orchestra.run.smithi168.stdout: "system_key": { 2020-11-29T15:57:11.397 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:57:11.397 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:57:11.398 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:11.398 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_pools": [ 2020-11-29T15:57:11.398 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:11.398 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "default-placement", 2020-11-29T15:57:11.398 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:57:11.399 INFO:teuthology.orchestra.run.smithi168.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-29T15:57:11.399 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": { 2020-11-29T15:57:11.399 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD": { 2020-11-29T15:57:11.399 INFO:teuthology.orchestra.run.smithi168.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-11-29T15:57:11.399 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:11.400 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:11.400 INFO:teuthology.orchestra.run.smithi168.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-29T15:57:11.400 INFO:teuthology.orchestra.run.smithi168.stdout: "index_type": 0 2020-11-29T15:57:11.400 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:11.400 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:11.401 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:11.401 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:11.401 INFO:teuthology.orchestra.run.smithi168.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-29T15:57:11.401 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:57:11.402 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-11-29T15:57:12.273 INFO:teuthology.orchestra.run.smithi168.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-11-29T15:57:12.291 INFO:teuthology.orchestra.run.smithi168:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-11-29T15:57:13.289 INFO:teuthology.orchestra.run.smithi168.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-11-29T15:57:13.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:13.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:13.314 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:57:13.358 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:13.358 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:13.380 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:57:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T15:57:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone3", 2020-11-29T15:57:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-11-29T15:57:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: "control_pool": "test-zone3.rgw.control", 2020-11-29T15:57:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-11-29T15:57:13.382 INFO:teuthology.orchestra.run.smithi168.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-11-29T15:57:13.382 INFO:teuthology.orchestra.run.smithi168.stdout: "log_pool": "test-zone3.rgw.log", 2020-11-29T15:57:13.382 INFO:teuthology.orchestra.run.smithi168.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-11-29T15:57:13.382 INFO:teuthology.orchestra.run.smithi168.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-11-29T15:57:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-11-29T15:57:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-11-29T15:57:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-11-29T15:57:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-11-29T15:57:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-11-29T15:57:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-11-29T15:57:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-11-29T15:57:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "system_key": { 2020-11-29T15:57:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:57:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:57:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_pools": [ 2020-11-29T15:57:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "default-placement", 2020-11-29T15:57:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:57:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-11-29T15:57:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": { 2020-11-29T15:57:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD": { 2020-11-29T15:57:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-11-29T15:57:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: "compression_type": "random" 2020-11-29T15:57:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-11-29T15:57:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: "index_type": 0 2020-11-29T15:57:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "", 2020-11-29T15:57:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-11-29T15:57:13.389 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:57:13.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:13.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:13.390 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:57:13.427 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:13.427 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:21.522 INFO:teuthology.orchestra.run.smithi168.stderr:Sending period to new master zone a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T15:57:21.589 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:57:21.589 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:21.590 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 3, 2020-11-29T15:57:21.590 INFO:teuthology.orchestra.run.smithi168.stdout: "predecessor_uuid": "f85496da-377c-48c0-b59c-86bb7b911fb2", 2020-11-29T15:57:21.590 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": [], 2020-11-29T15:57:21.590 INFO:teuthology.orchestra.run.smithi168.stdout: "period_map": { 2020-11-29T15:57:21.590 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:21.591 INFO:teuthology.orchestra.run.smithi168.stdout: "zonegroups": [ 2020-11-29T15:57:21.591 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:21.591 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "test-zonegroup", 2020-11-29T15:57:21.591 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zonegroup", 2020-11-29T15:57:21.592 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "", 2020-11-29T15:57:21.592 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "true", 2020-11-29T15:57:21.592 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T15:57:21.592 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:57:21.592 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:21.593 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T15:57:21.593 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T15:57:21.593 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:21.593 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [ 2020-11-29T15:57:21.593 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:21.594 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T15:57:21.594 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone3", 2020-11-29T15:57:21.594 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T15:57:21.594 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T15:57:21.595 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:21.595 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T15:57:21.595 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T15:57:21.595 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:21.595 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T15:57:21.596 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T15:57:21.596 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T15:57:21.596 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T15:57:21.596 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T15:57:21.596 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:21.597 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:21.597 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T15:57:21.597 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone2", 2020-11-29T15:57:21.597 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T15:57:21.597 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T15:57:21.598 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:21.598 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T15:57:21.598 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T15:57:21.598 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:21.598 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T15:57:21.599 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T15:57:21.599 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T15:57:21.599 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T15:57:21.599 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T15:57:21.599 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:21.600 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:21.600 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:21.600 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T15:57:21.600 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T15:57:21.600 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:57:21.601 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:21.601 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T15:57:21.601 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T15:57:21.601 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:21.602 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T15:57:21.602 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T15:57:21.602 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T15:57:21.602 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T15:57:21.602 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T15:57:21.603 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:21.603 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:21.603 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [ 2020-11-29T15:57:21.603 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:21.603 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default-placement", 2020-11-29T15:57:21.603 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-11-29T15:57:21.604 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": [ 2020-11-29T15:57:21.604 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD" 2020-11-29T15:57:21.604 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:57:21.604 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:21.604 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:21.605 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "default-placement", 2020-11-29T15:57:21.605 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:21.605 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T15:57:21.605 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T15:57:21.605 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:21.606 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:21.606 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:21.606 INFO:teuthology.orchestra.run.smithi168.stdout: "short_zone_ids": [ 2020-11-29T15:57:21.606 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:21.606 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T15:57:21.607 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 791374490 2020-11-29T15:57:21.607 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:21.607 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:21.607 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T15:57:21.607 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 2491292681 2020-11-29T15:57:21.608 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:21.608 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:21.608 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:21.608 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 1785095687 2020-11-29T15:57:21.608 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:21.609 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:57:21.609 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:21.609 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T15:57:21.609 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:21.609 INFO:teuthology.orchestra.run.smithi168.stdout: "period_config": { 2020-11-29T15:57:21.610 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_quota": { 2020-11-29T15:57:21.610 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T15:57:21.610 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T15:57:21.610 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T15:57:21.610 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T15:57:21.611 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T15:57:21.611 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:21.611 INFO:teuthology.orchestra.run.smithi168.stdout: "user_quota": { 2020-11-29T15:57:21.611 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T15:57:21.611 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T15:57:21.617 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T15:57:21.617 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T15:57:21.618 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T15:57:21.618 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:21.618 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:21.618 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:21.619 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_name": "test-realm", 2020-11-29T15:57:21.619 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:57:21.619 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:57:21.620 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-29T15:57:21.621 INFO:teuthology.orchestra.run:waiting for 300 2020-11-29T15:57:21.662 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T15:57:21.665+0000 7fbc76ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 14792) UID: 0 2020-11-29T15:57:21.663 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T15:57:21.665+0000 7fbc92c910c0 -1 shutting down 2020-11-29T15:57:26.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-29T15:57:27.724 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-29T15:57:27.724 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-29T15:57:27.725 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-29T15:57:27.727 INFO:tasks.rgw.c1.client.1:Started 2020-11-29T15:57:27.728 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-11-29T15:57:27.746 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:57:27.747 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:57:27.750 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi168.front.sepia.ceph.com port 8001: Connection refused 2020-11-29T15:57:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-29T15:57:28.752 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-11-29T15:57:28.768 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:57:28.768 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:57:28.775 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 30571 0 --:--:-- --:--:-- --:--:-- 30571 2020-11-29T15:57:28.775 INFO:teuthology.orchestra.run.smithi168.stdout:anonymous 2020-11-29T15:57:28.776 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-11-29T15:57:28.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi168.front.sepia.ceph.com:8000', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--url', 'http://smithi168.front.sepia.ceph.com:8000', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:28.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi168.front.sepia.ceph.com:8000', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--url', 'http://smithi168.front.sepia.ceph.com:8000', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:28.777 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi168.front.sepia.ceph.com:8000 --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --url http://smithi168.front.sepia.ceph.com:8000 --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T15:57:28.826 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:28.827 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:28.872 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T15:57:28.872+0000 7fc8957d2700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.a2a72fe6-27cf-4286-bdf9-88ee6023d0e2.control err=-107 2020-11-29T15:57:28.895 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:57:28.895 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:28.895 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-realm", 2020-11-29T15:57:28.896 INFO:teuthology.orchestra.run.smithi184.stdout: "current_period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:28.896 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2 2020-11-29T15:57:28.896 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:57:28.897 INFO:tasks.rgw_multisite:creating zone.. 2020-11-29T15:57:28.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:28.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:28.899 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --endpoints http://smithi184.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T15:57:28.950 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:28.951 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:29.016 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:57:29.016+0000 7f904bee2080 0 failed reading obj info from .rgw.root:zone_info.36679b47-164f-436b-b728-5846b32cea58: (2) No such file or directory 2020-11-29T15:57:29.017 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:57:29.016+0000 7f904bee2080 0 WARNING: could not read zone params for zone id=36679b47-164f-436b-b728-5846b32cea58 name=test-zone3 2020-11-29T15:57:29.019 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:57:29.020+0000 7f904bee2080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T15:57:29.019 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T15:57:29.020+0000 7f904bee2080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T15:57:29.027 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:57:29.027 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T15:57:29.028 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T15:57:29.028 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-29T15:57:29.028 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-29T15:57:29.028 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-29T15:57:29.029 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-29T15:57:29.029 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-29T15:57:29.029 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-29T15:57:29.029 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-29T15:57:29.029 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-29T15:57:29.030 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-29T15:57:29.030 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-29T15:57:29.030 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-29T15:57:29.030 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-29T15:57:29.031 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-29T15:57:29.031 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-29T15:57:29.031 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T15:57:29.031 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:57:29.031 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:57:29.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:29.032 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T15:57:29.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:29.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T15:57:29.032 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:29.033 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-29T15:57:29.033 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T15:57:29.033 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T15:57:29.033 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-11-29T15:57:29.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-29T15:57:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T15:57:29.034 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_config": { 2020-11-29T15:57:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "event_retention_days": 7, 2020-11-29T15:57:29.035 INFO:teuthology.orchestra.run.smithi184.stdout: "start_with_full_sync": false 2020-11-29T15:57:29.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:29.036 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:29.036 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-29T15:57:29.036 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:57:29.037 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-11-29T15:57:29.797 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-11-29T15:57:29.817 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-11-29T15:57:30.826 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-11-29T15:57:30.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:30.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:30.845 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T15:57:30.894 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:30.895 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:30.919 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:57:30.919 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T15:57:30.920 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T15:57:30.920 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-11-29T15:57:30.920 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone4.rgw.control", 2020-11-29T15:57:30.920 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-11-29T15:57:30.921 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-11-29T15:57:30.921 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone4.rgw.log", 2020-11-29T15:57:30.921 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-11-29T15:57:30.921 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-11-29T15:57:30.921 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-11-29T15:57:30.922 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-11-29T15:57:30.922 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-11-29T15:57:30.922 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-11-29T15:57:30.922 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-11-29T15:57:30.923 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-11-29T15:57:30.923 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-11-29T15:57:30.923 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T15:57:30.923 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T15:57:30.923 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T15:57:30.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:30.924 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T15:57:30.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:30.924 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T15:57:30.924 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:30.925 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-11-29T15:57:30.925 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T15:57:30.925 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T15:57:30.925 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-11-29T15:57:30.925 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T15:57:30.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:30.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:30.926 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-11-29T15:57:30.926 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T15:57:30.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:30.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:30.927 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:30.927 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_config": { 2020-11-29T15:57:30.927 INFO:teuthology.orchestra.run.smithi184.stdout: "event_retention_days": 7, 2020-11-29T15:57:30.928 INFO:teuthology.orchestra.run.smithi184.stdout: "start_with_full_sync": false 2020-11-29T15:57:30.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:30.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2020-11-29T15:57:30.928 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-11-29T15:57:30.928 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:57:30.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:30.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:30.930 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T15:57:30.968 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:30.968 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:31.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-29T15:57:39.030 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T15:57:39.102 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:57:39.102 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 4, 2020-11-29T15:57:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f85496da-377c-48c0-b59c-86bb7b911fb2", 2020-11-29T15:57:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [], 2020-11-29T15:57:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-11-29T15:57:39.104 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:39.104 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-11-29T15:57:39.104 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.104 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-11-29T15:57:39.104 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-11-29T15:57:39.105 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-11-29T15:57:39.105 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-11-29T15:57:39.105 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T15:57:39.105 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:57:39.106 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.106 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-11-29T15:57:39.106 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-11-29T15:57:39.106 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:39.106 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-11-29T15:57:39.107 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.107 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T15:57:39.107 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-11-29T15:57:39.107 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T15:57:39.107 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T15:57:39.108 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.108 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T15:57:39.108 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T15:57:39.108 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:39.108 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T15:57:39.109 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T15:57:39.109 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T15:57:39.109 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T15:57:39.109 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T15:57:39.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.110 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T15:57:39.110 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T15:57:39.110 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T15:57:39.111 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T15:57:39.111 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.111 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T15:57:39.111 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T15:57:39.112 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:39.112 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T15:57:39.112 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T15:57:39.112 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T15:57:39.113 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T15:57:39.113 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T15:57:39.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.113 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.114 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T15:57:39.114 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T15:57:39.114 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T15:57:39.114 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T15:57:39.114 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.115 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T15:57:39.115 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T15:57:39.115 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:39.115 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T15:57:39.115 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-11-29T15:57:39.116 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T15:57:39.116 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-11-29T15:57:39.116 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-11-29T15:57:39.116 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.116 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T15:57:39.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.117 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.117 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:39.117 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-11-29T15:57:39.117 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T15:57:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T15:57:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T15:57:39.118 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T15:57:39.119 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T15:57:39.119 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T15:57:39.119 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T15:57:39.119 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T15:57:39.119 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T15:57:39.120 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T15:57:39.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:39.120 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.121 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-11-29T15:57:39.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.121 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-11-29T15:57:39.121 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-11-29T15:57:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-11-29T15:57:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-11-29T15:57:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:57:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-11-29T15:57:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-11-29T15:57:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-11-29T15:57:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T15:57:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-11-29T15:57:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.281 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T15:57:39.281 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 791374490 2020-11-29T15:57:39.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T15:57:39.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2491292681 2020-11-29T15:57:39.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.283 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T15:57:39.283 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 293306859 2020-11-29T15:57:39.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:39.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:39.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1785095687 2020-11-29T15:57:39.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:39.284 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:57:39.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.285 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T15:57:39.285 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T15:57:39.285 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-11-29T15:57:39.285 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-11-29T15:57:39.285 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T15:57:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T15:57:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T15:57:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T15:57:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T15:57:39.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-11-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T15:57:39.287 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T15:57:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T15:57:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T15:57:39.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-11-29T15:57:39.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:39.289 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:57:39.290 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-11-29T15:57:39.290 INFO:teuthology.orchestra.run:waiting for 300 2020-11-29T15:57:39.333 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T15:57:39.336+0000 7fd9ae6a5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 14805) UID: 0 2020-11-29T15:57:39.333 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T15:57:39.336+0000 7fd9c61580c0 -1 shutting down 2020-11-29T15:57:44.764 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-11-29T15:57:45.392 INFO:tasks.rgw.c2.client.1:Stopped 2020-11-29T15:57:45.393 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-11-29T15:57:45.394 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-11-29T15:57:45.396 INFO:tasks.rgw.c2.client.1:Started 2020-11-29T15:57:45.397 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8001 2020-11-29T15:57:45.414 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:57:45.414 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:57:45.418 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8001: Connection refused 2020-11-29T15:57:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-29T15:57:46.420 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8001 2020-11-29T15:57:46.436 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T15:57:46.437 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-11-29T15:57:46.440 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 33500 2020-11-29T15:57:46.441 INFO:teuthology.orchestra.run.smithi184.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005fc3c4fa-10e5-test-zone4","HostId":"10e5-test-zone4-test-zonegroup"} 2020-11-29T15:57:46.442 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-11-29T15:57:46.442 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-11-29T15:57:46.507 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-11-29T15:57:46.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:46.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T15:57:46.508 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T15:57:46.553 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:46.554 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:46.774 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:57:46.775 INFO:teuthology.orchestra.run.smithi168.stdout: "user_id": "rgw-multisite-test-user", 2020-11-29T15:57:46.775 INFO:teuthology.orchestra.run.smithi168.stdout: "display_name": "Multisite Test User", 2020-11-29T15:57:46.775 INFO:teuthology.orchestra.run.smithi168.stdout: "email": "", 2020-11-29T15:57:46.775 INFO:teuthology.orchestra.run.smithi168.stdout: "suspended": 0, 2020-11-29T15:57:46.776 INFO:teuthology.orchestra.run.smithi168.stdout: "max_buckets": 1000, 2020-11-29T15:57:46.776 INFO:teuthology.orchestra.run.smithi168.stdout: "subusers": [], 2020-11-29T15:57:46.776 INFO:teuthology.orchestra.run.smithi168.stdout: "keys": [ 2020-11-29T15:57:46.776 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:46.777 INFO:teuthology.orchestra.run.smithi168.stdout: "user": "rgw-multisite-test-user", 2020-11-29T15:57:46.777 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "OFYG52MZYBDXJQWZ56IL", 2020-11-29T15:57:46.777 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "AWlOqyUNZDlNGroUVewOVQycnZ03yi83fL54nd78" 2020-11-29T15:57:46.777 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:46.778 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T15:57:46.778 INFO:teuthology.orchestra.run.smithi168.stdout: "swift_keys": [], 2020-11-29T15:57:46.778 INFO:teuthology.orchestra.run.smithi168.stdout: "caps": [], 2020-11-29T15:57:46.778 INFO:teuthology.orchestra.run.smithi168.stdout: "op_mask": "read, write, delete", 2020-11-29T15:57:46.778 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "", 2020-11-29T15:57:46.779 INFO:teuthology.orchestra.run.smithi168.stdout: "default_storage_class": "", 2020-11-29T15:57:46.779 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_tags": [], 2020-11-29T15:57:46.779 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_quota": { 2020-11-29T15:57:46.779 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T15:57:46.779 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T15:57:46.780 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T15:57:46.780 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T15:57:46.780 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T15:57:46.780 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:46.780 INFO:teuthology.orchestra.run.smithi168.stdout: "user_quota": { 2020-11-29T15:57:46.781 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T15:57:46.781 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T15:57:46.781 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T15:57:46.781 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T15:57:46.782 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T15:57:46.782 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:46.782 INFO:teuthology.orchestra.run.smithi168.stdout: "temp_url_keys": [], 2020-11-29T15:57:46.782 INFO:teuthology.orchestra.run.smithi168.stdout: "type": "rgw", 2020-11-29T15:57:46.782 INFO:teuthology.orchestra.run.smithi168.stdout: "mfa_ids": [] 2020-11-29T15:57:46.783 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:57:46.783 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T15:57:46.804 INFO:rgw_multi.tests:meta checkpoint 2020-11-29T15:57:46.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:57:46.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:57:46.806 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:57:46.844 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:46.845 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:47.062 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T15:57:47.095 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T15:57:47.096 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:47.096 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665466.770600_48.1", 2020-11-29T15:57:47.096 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:57:46.770600Z" 2020-11-29T15:57:47.096 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:47.096 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:47.097 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:57:47.097 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:57:47.097 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:47.097 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:47.098 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:57:47.098 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:57:47.098 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:57:47.098 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:57:47.098 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:57:47.099 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:57:47.099 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:57:47.099 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T15:57:47.100 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T15:57:47.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:57:47.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:57:47.101 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:57:47.142 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:47.142 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:47.403 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:57:47.404 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:57:47.404 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:57:47.405 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:57:47.405 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:57:47.406 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:47.407 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:47.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:47.407 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:57:47.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:47.408 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:57:47.409 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:47.409 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:47.410 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:47.410 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:47.411 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:57:47.411 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:47.412 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:47.412 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:47.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:47.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:47.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:47.414 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:57:47.414 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:47.415 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:47.415 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:47.416 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:47.416 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:47.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:47.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:47.418 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:47.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:47.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:47.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:47.420 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:57:47.420 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:47.421 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:47.421 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:47.422 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:47.422 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:47.422 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:47.423 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:47.423 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:47.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:47.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:47.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:47.425 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:57:47.426 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:47.426 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:47.427 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:47.427 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:47.428 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:47.428 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:47.429 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:47.429 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:47.429 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:47.430 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:47.430 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:57:47.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:47.431 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:57:47.432 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:57:47.432 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:57:47.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:47.433 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:57:47.435 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:57:50.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-11-29T15:57:52.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:57:52.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:57:52.438 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:57:52.480 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:52.480 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:52.909 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:57:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:57:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:57:52.910 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:57:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:57:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:52.911 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:57:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:57:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:52.912 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:52.913 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:57:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:52.914 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:57:52.915 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:52.916 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:52.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:57:52.918 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:52.919 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:52.920 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:52.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:57:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:52.922 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:52.923 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:52.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:57:52.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:57:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:57:52.926 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:57:52.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:52.927 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:57:52.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:57:57.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:57:57.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:57:57.930 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:57:57.970 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:57:57.970 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:57:58.217 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:57:58.217 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:57:58.217 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:57:58.218 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:57:58.218 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:57:58.218 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:57:58.218 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:58.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:58.219 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:57:58.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:58.219 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:57:58.220 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:58.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:58.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:58.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:58.220 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:57:58.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:58.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:58.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:58.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:58.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:58.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:58.222 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:57:58.222 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:58.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:58.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:58.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:58.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:58.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:58.226 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:58.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:57:58.228 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:57:58.229 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:57:58.230 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:57:58.231 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:57:58.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:57:58.232 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:57:58.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:57:58.232 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:57:58.234 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:03.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:03.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:03.235 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:03.278 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:03.279 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:03.534 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:03.535 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:03.535 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:03.535 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:03.535 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:03.536 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:03.536 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:03.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:03.537 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:03.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:03.537 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:03.537 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:03.538 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:03.538 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:03.538 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:03.538 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:03.539 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:03.539 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:03.539 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:03.539 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:03.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:03.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:03.540 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:03.540 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:03.540 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:03.541 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:03.541 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:03.541 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:03.541 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:03.542 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:03.542 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:03.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:03.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:03.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:03.543 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:03.544 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:03.544 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:03.544 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:03.544 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:03.544 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:03.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:03.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:03.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:03.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:03.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:03.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:03.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:03.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:03.547 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:03.547 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:03.547 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:03.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:03.548 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:03.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:03.548 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:03.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:03.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:03.549 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:03.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:03.549 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:03.549 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:03.550 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:03.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:03.550 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:03.551 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:08.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:08.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:08.554 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:08.594 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:08.594 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:08.854 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:08.855 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:08.855 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:08.855 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:08.856 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:08.856 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:08.856 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:08.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:08.856 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:08.857 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:08.857 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:08.857 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:08.857 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:08.858 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:08.858 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:08.858 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:08.858 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:08.858 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:08.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:08.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:08.861 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:08.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:08.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:08.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:08.861 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:08.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:08.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:08.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:08.862 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:08.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:08.863 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:08.863 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:08.863 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:08.863 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:08.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:08.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:08.865 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:08.865 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:08.865 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:08.865 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:08.866 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:08.866 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:08.866 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:08.866 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:08.867 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:08.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:08.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:08.867 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:08.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:08.868 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:08.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:08.868 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:08.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:08.868 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:08.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:13.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:13.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:13.872 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:13.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:13.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:14.166 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:14.180 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:14.181 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:19.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:19.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:19.182 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:19.222 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:19.474 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:19.475 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:19.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:19.477 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:19.478 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:19.478 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:19.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:19.478 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:19.479 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:19.480 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:19.481 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:19.481 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:19.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:19.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:19.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:19.482 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:19.482 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:19.482 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:19.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:19.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:19.483 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:19.483 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:19.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:19.483 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:19.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:19.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:19.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:19.484 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:19.484 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:19.484 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:19.485 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:19.485 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:19.485 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:19.485 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:19.485 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:19.486 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:19.486 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:19.486 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:19.486 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:19.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:19.487 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:19.487 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:19.487 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:19.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:19.487 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:19.488 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:24.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:24.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:24.491 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:24.532 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:24.533 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:24.798 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:24.798 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:24.798 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:24.799 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:24.799 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:24.799 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:24.799 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:24.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:24.800 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:24.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:24.800 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:24.800 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:24.801 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:24.801 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:24.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:24.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:24.801 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:24.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:24.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:24.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:24.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:24.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:24.803 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:24.803 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:24.803 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:24.803 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:24.804 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:24.804 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:24.804 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:24.804 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:24.804 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:24.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:24.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:24.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:24.805 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:24.805 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:24.806 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:24.806 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:24.806 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:24.806 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:24.807 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:24.807 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:24.807 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:24.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:24.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:24.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:24.808 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:24.808 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:24.808 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:24.808 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:24.809 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:24.809 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:24.809 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:24.809 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:24.810 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:24.810 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:24.810 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:24.810 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:24.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:24.811 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:24.811 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:24.811 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:24.811 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:24.811 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:24.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:29.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:29.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:29.814 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:29.853 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:29.854 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:30.088 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:30.089 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:30.089 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:30.089 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:30.089 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:30.090 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:30.090 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:30.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:30.091 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:30.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:30.091 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:30.091 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:30.091 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:30.092 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:30.092 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:30.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:30.092 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:30.093 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:30.093 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:30.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:30.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:30.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:30.094 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:30.094 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:30.094 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:30.094 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:30.095 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:30.095 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:30.095 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:30.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:30.095 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:30.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:30.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:30.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:30.096 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:30.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:30.097 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:30.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:30.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:30.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:30.098 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:30.098 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:30.098 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:30.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:30.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:30.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:30.099 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:30.099 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:30.099 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:30.100 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:30.100 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:30.100 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:30.100 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:30.101 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:30.101 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:30.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:30.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:30.101 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:30.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:30.102 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:30.102 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:30.102 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:30.103 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:30.103 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:30.104 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:35.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:35.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:35.106 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:35.146 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:35.146 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:35.394 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:35.395 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:35.396 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:35.396 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:35.396 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:35.397 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:35.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:35.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:35.398 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:35.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:35.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:35.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:35.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:35.401 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:35.402 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:35.402 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:35.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:35.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:35.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:35.403 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:35.403 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:35.403 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:35.403 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:35.404 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:35.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:35.404 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:35.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:35.404 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:35.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:35.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:35.405 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:35.405 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:35.405 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:35.406 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:35.406 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:35.406 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:35.406 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:35.407 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:35.407 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:35.407 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:35.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:35.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:35.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:35.408 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:35.408 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:35.408 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:35.408 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:35.409 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:35.409 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:35.409 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:35.409 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:35.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:35.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:35.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:35.410 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:35.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:35.410 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:35.411 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:35.411 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:35.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:35.411 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:35.412 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:40.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:40.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:40.418 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:40.458 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:40.458 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:40.713 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:40.714 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:40.714 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:40.714 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:40.715 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:40.715 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:40.715 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:40.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:40.715 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:40.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:40.716 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:40.716 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:40.716 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:40.716 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:40.717 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:40.717 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:40.717 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:40.717 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:40.718 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:40.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:40.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:40.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:40.718 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:40.719 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:40.719 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:40.719 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:40.719 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:40.719 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:40.720 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:40.720 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:40.720 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:40.720 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:40.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:40.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:40.721 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:40.721 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:40.721 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:40.721 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:40.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:40.722 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:40.722 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:40.722 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:40.723 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:40.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:40.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:40.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:40.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:40.724 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:40.724 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:40.724 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:40.724 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:40.724 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:40.725 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:40.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:40.725 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:40.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:40.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:40.726 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:40.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:40.726 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:40.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:40.726 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:40.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:40.727 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:40.728 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:45.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:45.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:45.731 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:45.772 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:45.772 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:46.017 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:46.018 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:46.018 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:46.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:46.021 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:46.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:46.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:46.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:46.024 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:46.024 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:46.024 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:46.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:46.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:46.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:46.025 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:46.025 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:46.025 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:46.026 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:46.026 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:46.026 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:46.026 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:46.026 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:46.027 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:46.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:46.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:46.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:46.027 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:46.028 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:46.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:46.030 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:46.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:46.030 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:46.030 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:46.031 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:46.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:46.031 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:46.032 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:51.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:51.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:51.034 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:51.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:51.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:51.334 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:51.334 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:51.335 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:51.335 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:51.335 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:51.335 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:51.336 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:51.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:51.336 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:51.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:51.337 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:51.337 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:51.337 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:51.337 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:51.338 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:51.338 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:51.338 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:51.338 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:51.339 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:51.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:51.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:51.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:51.340 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:51.340 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:51.340 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:51.340 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:51.341 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:51.341 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:51.341 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:51.341 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:51.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:51.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:51.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:51.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:51.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:51.343 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:51.343 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:51.343 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:51.343 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:51.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:51.344 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:51.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:51.344 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:51.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:51.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:51.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:51.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:51.346 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:51.346 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:51.346 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:51.346 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:51.347 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:51.347 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:51.347 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:51.347 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:51.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:51.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:51.348 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:51.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:51.350 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:51.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:58:56.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:56.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:58:56.353 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:58:56.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:58:56.394 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:58:56.650 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:58:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:58:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:58:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:58:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:58:56.651 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:56.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:56.652 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:58:56.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:56.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:58:56.653 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:56.653 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:56.653 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:56.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:56.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:58:56.654 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:56.654 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:56.654 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:56.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:56.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:56.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:56.655 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:58:56.655 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:56.655 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:56.655 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:56.656 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:56.657 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:56.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:56.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:56.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:56.657 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:58:56.658 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:56.658 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:56.658 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:56.658 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:56.658 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:56.659 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:56.659 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:56.659 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:56.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:56.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:56.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:58:56.660 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:58:56.660 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:58:56.660 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:58:56.661 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:58:56.661 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:58:56.661 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:58:56.661 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:58:56.661 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:58:56.662 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:58:56.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:56.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:56.662 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:58:56.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:58:56.663 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:58:56.663 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:58:56.663 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:58:56.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:58:56.664 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:58:56.665 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:01.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:01.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:01.667 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:01.708 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:01.709 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:01.961 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:01.961 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:01.961 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:01.962 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:01.962 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:01.962 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:01.962 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:01.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:01.963 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:01.964 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:01.965 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:01.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:01.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:01.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:01.966 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:01.966 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:01.966 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:01.966 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:01.966 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:01.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:01.968 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:01.969 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:01.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:01.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:01.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:01.973 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:01.974 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:01.974 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:01.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:01.974 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:01.975 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:06.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:06.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:06.977 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:07.018 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:07.019 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:07.265 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:07.265 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:07.265 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:07.265 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:07.266 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:07.266 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:07.266 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:07.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:07.267 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:07.267 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:07.267 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:07.267 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:07.268 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:07.268 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:07.268 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:07.268 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:07.269 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:07.269 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:07.269 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:07.269 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:07.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:07.270 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:07.270 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:07.270 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:07.270 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:07.271 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:07.271 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:07.271 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:07.271 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:07.271 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:07.272 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:07.272 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:07.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:07.272 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:07.272 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:07.273 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:07.273 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:07.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:07.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:07.273 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:07.274 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:07.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:07.274 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:07.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:07.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:07.277 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:07.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:07.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:07.277 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:07.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:07.278 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:07.278 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:07.278 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:07.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:07.278 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:07.279 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:12.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:12.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:12.281 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:12.321 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:12.321 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:12.589 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:12.590 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:12.590 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:12.590 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:12.590 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:12.591 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:12.591 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:12.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:12.591 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:12.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:12.592 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:12.592 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:12.592 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:12.592 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:12.592 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:12.593 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:12.593 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:12.593 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:12.593 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:12.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:12.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:12.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:12.594 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:12.594 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:12.595 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:12.595 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:12.595 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:12.595 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:12.595 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:12.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:12.596 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:12.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:12.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:12.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:12.597 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:12.597 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:12.597 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:12.597 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:12.598 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:12.598 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:12.598 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:12.598 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:12.599 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:12.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:12.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:12.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:12.599 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:12.600 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:12.600 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:12.600 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:12.600 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:12.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:12.601 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:12.601 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:12.601 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:12.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:12.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:12.602 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:12.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:12.602 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:12.602 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:12.602 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:12.603 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:12.603 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:12.604 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:17.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:17.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:17.606 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:17.648 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:17.648 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:17.907 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:17.907 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:17.908 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:17.909 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:17.910 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:17.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:17.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:17.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:17.912 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:17.912 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:17.912 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:17.913 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:17.913 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:17.913 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:17.913 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:17.913 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:17.914 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:17.914 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:17.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:17.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:17.915 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:17.915 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:17.915 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:17.915 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:17.915 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:17.916 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:17.916 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:17.916 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:17.916 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:17.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:17.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:17.917 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:17.917 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:17.917 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:17.918 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:17.918 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:17.918 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:17.918 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:17.918 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:17.919 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:17.919 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:17.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:17.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:17.920 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:17.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:17.920 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:17.920 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:17.920 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:17.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:17.921 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:17.922 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:22.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:22.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:22.924 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:22.964 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:22.965 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:23.218 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:23.218 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:23.218 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:23.218 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:23.219 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:23.219 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:23.219 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:23.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:23.219 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:23.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:23.220 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:23.220 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:23.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:23.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:23.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:23.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:23.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:23.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:23.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:23.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:23.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:23.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:23.222 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:23.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:23.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:23.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:23.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:23.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:23.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:23.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:23.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:23.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:23.226 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:23.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:23.226 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:23.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:23.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:23.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:23.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:23.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:23.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:23.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:23.228 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:23.228 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:23.228 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:23.229 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:23.229 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:23.229 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:23.229 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:23.229 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:23.230 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:23.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:23.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:23.230 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:23.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:23.231 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:23.231 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:23.231 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:23.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:23.231 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:23.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:28.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:28.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:28.234 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:28.276 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:28.276 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:28.538 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:28.539 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:28.540 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:28.541 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:28.542 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:28.543 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:28.544 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:28.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:28.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:28.547 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:28.548 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:28.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:28.550 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:28.551 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:28.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:28.553 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:28.554 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:28.555 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:28.556 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:28.556 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:28.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:28.556 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:28.557 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:33.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:33.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:33.559 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:33.600 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:33.600 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:33.857 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:33.857 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:33.858 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:33.859 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:33.860 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:33.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:33.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:33.861 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:33.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:33.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:33.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:33.862 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:33.862 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:33.862 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:33.863 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:33.863 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:33.863 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:33.863 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:33.863 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:33.864 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:33.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:33.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:33.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:33.864 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:33.865 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:33.865 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:33.865 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:33.865 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:33.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:33.866 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:33.866 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:33.866 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:33.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:33.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:33.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:33.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:33.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:33.867 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:33.867 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:33.868 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:33.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:33.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:33.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:33.868 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:33.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:33.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:33.869 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:33.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:33.869 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:33.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:33.870 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:33.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:33.870 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:33.871 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:38.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:38.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:38.873 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:38.911 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:38.911 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:39.177 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:39.177 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:39.178 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:39.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:39.179 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:39.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:39.179 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:39.179 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:39.180 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:39.180 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:39.180 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:39.180 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:39.180 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:39.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:39.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:39.181 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:39.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:39.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:39.182 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:39.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:39.182 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:39.182 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:39.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:39.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:39.183 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:39.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:39.183 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:39.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:39.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:39.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:39.184 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:39.184 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:39.185 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:39.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:39.185 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:39.185 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:39.186 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:39.186 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:39.186 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:39.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:39.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:39.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:39.187 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:39.187 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:39.187 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:39.187 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:39.188 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:39.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:39.188 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:39.188 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:39.189 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:39.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:39.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:39.189 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:39.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:39.190 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:39.190 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:39.190 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:39.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:39.190 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:39.191 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1606665466.770600_48.1 target= 2020-11-29T15:59:44.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:44.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:44.193 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:44.235 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:44.236 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:44.486 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:44.487 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:44.487 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:44.487 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:44.487 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:44.488 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:44.488 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:44.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:44.488 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:44.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:44.489 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:44.489 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:44.489 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:44.489 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665466.770600_48.1", 2020-11-29T15:59:44.490 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:44.490 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:44.490 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:44.490 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:57:46.770600Z", 2020-11-29T15:59:44.490 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:44.491 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:44.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:44.491 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:44.491 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:44.491 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:44.492 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:44.492 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:44.492 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:44.492 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:44.493 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:44.493 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:44.493 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:44.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:44.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:44.494 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:44.495 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:44.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:44.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:44.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:44.496 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:44.496 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:44.497 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:44.497 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:44.497 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:44.497 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:44.497 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:44.498 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:44.498 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:44.498 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:44.498 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:44.498 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:44.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:44.499 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:44.499 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:44.499 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:44.500 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:44.500 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:44.501 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T15:59:44.501 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T15:59:44.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:44.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:44.502 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:44.542 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:44.543 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:44.797 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:59:44.798 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T15:59:44.798 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T15:59:44.798 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T15:59:44.798 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T15:59:44.799 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:44.799 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:44.799 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:44.799 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T15:59:44.799 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:44.800 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T15:59:44.800 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:44.800 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:44.800 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665466.770600_48.1", 2020-11-29T15:59:44.800 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:44.801 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T15:59:44.801 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:44.801 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:57:46.770600Z", 2020-11-29T15:59:44.801 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:44.802 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:44.802 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:44.802 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:44.802 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T15:59:44.802 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:44.803 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:44.803 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:44.803 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:44.803 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:44.803 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:44.804 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:44.804 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:44.804 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:44.804 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:44.804 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:44.805 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T15:59:44.805 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:44.805 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:44.805 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:44.805 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:44.806 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:44.806 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:44.806 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:44.806 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:44.806 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:44.807 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:44.807 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:44.807 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T15:59:44.807 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:44.807 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:44.808 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:44.808 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:44.808 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:44.808 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:44.809 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:44.809 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:44.809 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:44.809 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:44.809 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:59:44.810 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:44.810 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T15:59:44.810 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T15:59:44.810 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T15:59:44.810 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:44.811 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:59:44.812 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T15:59:44.812 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T15:59:44.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:44.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:44.813 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:44.853 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:44.854 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:45.118 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:45.119 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:45.119 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:45.119 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:45.120 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:45.120 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:45.120 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.120 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:45.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:45.121 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:45.121 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:45.121 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:45.121 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665466.770600_48.1", 2020-11-29T15:59:45.122 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:45.122 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:45.122 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:45.122 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:57:46.770600Z", 2020-11-29T15:59:45.123 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:45.123 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:45.124 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:45.124 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:45.124 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:45.124 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:45.124 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:45.125 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:45.125 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:45.125 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.126 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:45.126 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:45.126 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:45.126 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:45.127 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:45.127 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:45.127 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:45.127 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:45.127 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:45.128 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.128 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:45.128 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:45.129 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:45.130 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:45.130 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:45.130 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.131 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.131 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:45.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.131 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:45.131 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:45.132 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:45.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.132 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:45.133 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T15:59:45.133 INFO:rgw_multi.tests:meta checkpoint 2020-11-29T15:59:45.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:45.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:45.135 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:45.174 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:45.174 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:45.398 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T15:59:45.429 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T15:59:45.430 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:45.430 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665466.770600_48.1", 2020-11-29T15:59:45.430 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:57:46.770600Z" 2020-11-29T15:59:45.430 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:45.431 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:45.431 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:45.431 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:45.431 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:45.432 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:45.432 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:45.432 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:45.432 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:45.432 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:45.433 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:45.433 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:45.433 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:45.433 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T15:59:45.434 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T15:59:45.435 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:45.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:45.435 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:45.477 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:45.477 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:45.728 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:45.729 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:45.729 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:45.729 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:45.730 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:45.730 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:45.730 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.731 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:45.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:45.731 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:45.731 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:45.732 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:45.732 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665466.770600_48.1", 2020-11-29T15:59:45.732 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:45.732 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:45.732 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:45.733 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:57:46.770600Z", 2020-11-29T15:59:45.733 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:45.734 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:45.734 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:45.734 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:45.734 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:45.735 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:45.735 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:45.735 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:45.735 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:45.735 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.736 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:45.736 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:45.737 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:45.737 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:45.737 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:45.737 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:45.737 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:45.738 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:45.738 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:45.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:45.739 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:45.739 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:45.739 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:45.742 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:45.744 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T15:59:45.744 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T15:59:45.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:45.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:45.745 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:45.786 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:45.786 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:46.033 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:59:46.033 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T15:59:46.033 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T15:59:46.034 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T15:59:46.034 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T15:59:46.034 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:46.034 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:46.035 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:46.035 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T15:59:46.035 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:46.035 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T15:59:46.035 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:46.036 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:46.036 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665466.770600_48.1", 2020-11-29T15:59:46.036 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:46.036 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T15:59:46.036 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:46.037 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:57:46.770600Z", 2020-11-29T15:59:46.037 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:46.037 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:46.037 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:46.037 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:46.038 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T15:59:46.038 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:46.038 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:46.038 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:46.038 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:46.039 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:46.039 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:46.039 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:46.039 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:46.039 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:46.040 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:46.040 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:46.040 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T15:59:46.040 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:46.040 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:46.041 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:46.041 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:46.041 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:46.041 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:46.041 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:46.041 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:46.042 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:46.042 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:46.042 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:46.042 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T15:59:46.042 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:46.043 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:46.043 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:46.043 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:46.043 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:46.043 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:46.044 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:46.044 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:46.044 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:46.044 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:46.044 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:59:46.045 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:46.045 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T15:59:46.045 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T15:59:46.045 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T15:59:46.045 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:46.046 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:59:46.047 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T15:59:46.047 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T15:59:46.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:46.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:46.048 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:46.088 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:46.088 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:46.350 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:46.351 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:46.351 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:46.351 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:46.351 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:46.352 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:46.352 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:46.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:46.352 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:46.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:46.353 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:46.353 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:46.353 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:46.353 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665466.770600_48.1", 2020-11-29T15:59:46.354 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:46.354 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:46.354 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:46.354 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:57:46.770600Z", 2020-11-29T15:59:46.354 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:46.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:46.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:46.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:46.355 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:46.356 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:46.356 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:46.356 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:46.356 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:46.356 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:46.357 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:46.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:46.357 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:46.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:46.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:46.358 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:46.358 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:46.358 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:46.359 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:46.359 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:46.359 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:46.359 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:46.359 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:46.360 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:46.360 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:46.360 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:46.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:46.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:46.361 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:46.361 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:46.361 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:46.361 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:46.362 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:46.362 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:46.362 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:46.362 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:46.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:46.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:46.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:46.363 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:46.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:46.364 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:46.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:46.364 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:46.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:46.365 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:46.366 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T15:59:46.366 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-11-29T15:59:46.373 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-1 2020-11-29T15:59:50.332 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-2 2020-11-29T15:59:53.886 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-3 2020-11-29T15:59:53.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:53.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:53.947 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:53.979 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:53.980 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:54.188 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T15:59:54.222 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T15:59:54.222 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:54.223 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665593.929811_62.1", 2020-11-29T15:59:54.223 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:53.929811Z" 2020-11-29T15:59:54.223 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:54.223 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:54.224 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:54.224 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:54.224 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:54.224 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:54.225 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:54.225 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:54.225 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:54.225 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:54.225 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:54.226 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:54.226 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:54.226 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T15:59:54.227 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T15:59:54.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:54.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:54.229 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:54.268 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:54.268 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:54.534 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:54.534 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:54.534 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:54.535 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:54.535 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:54.535 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:54.535 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:54.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:54.536 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:54.536 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:54.536 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:54.536 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:54.537 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:54.537 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665593.929811_62.1", 2020-11-29T15:59:54.537 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:54.537 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:54.537 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:54.538 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.929811Z", 2020-11-29T15:59:54.538 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:54.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:54.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:54.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:54.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:54.539 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:54.539 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:54.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:54.540 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:54.540 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:54.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:54.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:54.540 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:54.541 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:54.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:54.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:54.541 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:54.542 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:54.542 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:54.542 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:54.542 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:54.542 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:54.543 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:54.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:54.543 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:54.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:54.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:54.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:54.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:54.544 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:54.544 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:54.544 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:54.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:54.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:54.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:54.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:54.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:54.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:54.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:54.546 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:54.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:54.547 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:54.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:54.547 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:54.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:54.547 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:54.548 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T15:59:54.549 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T15:59:54.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:54.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:54.550 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:54.592 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:54.592 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:54.848 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:59:54.848 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T15:59:54.848 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T15:59:54.849 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T15:59:54.849 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T15:59:54.849 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:54.849 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:54.850 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:54.850 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T15:59:54.850 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:54.850 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T15:59:54.851 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:54.851 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:54.851 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665593.929811_62.1", 2020-11-29T15:59:54.851 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:54.851 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T15:59:54.852 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:54.852 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.929811Z", 2020-11-29T15:59:54.852 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:54.852 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:54.852 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:54.853 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:54.853 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T15:59:54.853 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:54.853 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:54.854 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:54.854 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:54.854 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:54.854 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:54.854 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:54.855 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:54.855 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:54.855 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:54.855 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:54.855 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T15:59:54.856 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:54.856 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:54.856 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:54.856 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:54.857 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:54.857 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:54.857 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:54.857 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:54.857 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:54.858 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:54.858 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:54.858 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T15:59:54.858 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:54.859 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:54.859 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:54.859 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:54.859 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:54.859 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:54.860 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:54.860 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:54.860 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:54.860 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:54.861 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:59:54.861 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:54.861 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T15:59:54.861 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T15:59:54.861 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T15:59:54.862 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:54.862 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:59:54.863 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T15:59:54.864 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T15:59:54.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:54.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:54.865 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:54.904 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:54.904 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:55.171 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:55.171 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:55.172 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:55.173 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:55.173 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:55.173 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:55.173 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.174 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:55.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:55.174 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:55.175 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:55.175 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:55.175 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665593.929811_62.1", 2020-11-29T15:59:55.175 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:55.176 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:55.176 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:55.176 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.929811Z", 2020-11-29T15:59:55.176 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:55.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:55.177 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:55.178 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:55.178 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:55.178 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:55.178 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:55.179 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:55.179 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:55.180 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:55.181 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:55.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:55.182 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:55.182 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:55.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:55.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:55.183 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:55.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:55.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:55.184 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:55.185 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:55.185 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:55.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:55.185 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:55.186 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:55.186 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:55.186 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:55.186 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.187 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:55.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.187 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:55.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:55.188 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:55.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.188 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:55.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T15:59:55.234 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-11-29T15:59:55.237 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-4 2020-11-29T15:59:55.266 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-5 2020-11-29T15:59:55.325 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-6 2020-11-29T15:59:55.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:55.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:55.383 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:55.420 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:55.420 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:55.639 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T15:59:55.673 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T15:59:55.673 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:55.674 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665595.365399_75.1", 2020-11-29T15:59:55.674 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:55.365399Z" 2020-11-29T15:59:55.674 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:55.674 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:55.674 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:55.675 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:55.675 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:55.675 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:55.675 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:55.676 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:55.676 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:55.676 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:55.676 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:55.676 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:55.677 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:55.677 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T15:59:55.678 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T15:59:55.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:55.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:55.679 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:55.721 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:55.721 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:55.966 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:55.967 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:55.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:55.970 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:55.970 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665595.365399_75.1", 2020-11-29T15:59:55.970 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:55.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:55.971 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:55.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.365399Z", 2020-11-29T15:59:55.971 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:55.972 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:55.972 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:55.972 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:55.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:55.974 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:55.975 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:55.975 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:55.975 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:55.975 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:55.975 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:55.976 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:55.976 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:55.976 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:55.976 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.977 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.977 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:55.977 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:55.978 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:55.978 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:55.978 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:55.978 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:55.979 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:55.979 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:55.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:55.980 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:55.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.980 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:55.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:55.981 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:55.981 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:55.981 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:55.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:55.982 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:55.983 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T15:59:55.983 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T15:59:55.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:55.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:55.985 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:56.025 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:56.025 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:56.274 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:59:56.275 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T15:59:56.275 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T15:59:56.275 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T15:59:56.275 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T15:59:56.276 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:56.276 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:56.276 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:56.276 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T15:59:56.276 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:56.277 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T15:59:56.277 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:56.277 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:56.277 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665595.365399_75.1", 2020-11-29T15:59:56.277 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:56.278 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T15:59:56.278 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:56.278 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.365399Z", 2020-11-29T15:59:56.279 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:56.279 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:56.279 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:56.280 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:56.280 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T15:59:56.280 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:56.280 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:56.280 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:56.281 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:56.281 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:56.281 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:56.281 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:56.282 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:56.282 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:56.282 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:56.282 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:56.283 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T15:59:56.283 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:56.284 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:56.284 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:56.284 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:56.285 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:56.285 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:56.286 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:56.286 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:56.286 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:56.287 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:56.287 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:56.288 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T15:59:56.288 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:56.288 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:56.289 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:56.289 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:56.290 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:56.290 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:56.291 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:56.291 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:56.291 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:56.292 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:56.292 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:59:56.293 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:56.293 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T15:59:56.293 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T15:59:56.294 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T15:59:56.294 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:56.295 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:59:56.296 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T15:59:56.297 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T15:59:56.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:56.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:56.300 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:56.340 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:56.340 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:56.604 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:56.605 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:56.605 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:56.605 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:56.605 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:56.606 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:56.607 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665595.365399_75.1", 2020-11-29T15:59:56.608 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:56.608 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:56.608 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:56.608 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.365399Z", 2020-11-29T15:59:56.608 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:56.609 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:56.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:56.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:56.609 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:56.609 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:56.610 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:56.610 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:56.610 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:56.610 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:56.610 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:56.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:56.611 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:56.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:56.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:56.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:56.612 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:56.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:56.612 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:56.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:56.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:56.613 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:56.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:56.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:56.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:56.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:56.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:56.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:56.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:56.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:56.614 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:56.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:56.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:56.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:56.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:56.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:56.616 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:56.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:56.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:56.616 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:56.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:56.617 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:56.617 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:56.617 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:56.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:56.617 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:56.618 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T15:59:56.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:56.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:56.961 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:56.993 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:56.993 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:57.204 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T15:59:57.239 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T15:59:57.239 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:57.239 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665595.365399_75.1", 2020-11-29T15:59:57.239 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:55.365399Z" 2020-11-29T15:59:57.240 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:57.240 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:57.240 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:57.240 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:57.240 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:57.241 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:57.241 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:57.241 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:57.241 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:57.241 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:57.242 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:57.242 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:57.242 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:57.242 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T15:59:57.243 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T15:59:57.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:57.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:57.244 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:57.285 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:57.285 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:57.536 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:57.537 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:57.537 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:57.537 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:57.538 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:57.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665595.365399_75.1", 2020-11-29T15:59:57.540 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:57.540 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:57.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:57.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.365399Z", 2020-11-29T15:59:57.540 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:57.541 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:57.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:57.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:57.541 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:57.541 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:57.542 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:57.542 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:57.542 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:57.542 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:57.542 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:57.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:57.543 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:57.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:57.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:57.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:57.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:57.544 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:57.544 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:57.544 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:57.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:57.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:57.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:57.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:57.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:57.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:57.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:57.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:57.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:57.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:57.547 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:57.547 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:57.547 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:57.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:57.547 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:57.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:57.548 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:57.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:57.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:57.548 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:57.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:57.549 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:57.549 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:57.549 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:57.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:57.550 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:57.551 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T15:59:57.552 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T15:59:57.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:57.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:57.554 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:57.594 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:57.594 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:57.857 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:59:57.858 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T15:59:57.858 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T15:59:57.858 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T15:59:57.858 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T15:59:57.859 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:57.859 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:57.859 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:57.859 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T15:59:57.859 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:57.860 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T15:59:57.860 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:57.860 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:57.860 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665595.365399_75.1", 2020-11-29T15:59:57.860 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:57.861 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T15:59:57.861 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:57.861 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.365399Z", 2020-11-29T15:59:57.861 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:57.861 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:57.862 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:57.862 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:57.862 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T15:59:57.862 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:57.862 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:57.863 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:57.863 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:57.863 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:57.863 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:57.863 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:57.864 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:57.864 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:57.864 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:57.864 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:57.865 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T15:59:57.865 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:57.865 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:57.865 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:57.865 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:57.866 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:57.866 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:57.866 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:57.866 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:57.867 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:57.867 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:57.867 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:57.867 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T15:59:57.867 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:57.868 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:57.868 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:57.868 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:57.868 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:57.869 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:57.869 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:57.869 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:57.869 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:57.869 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:57.870 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:59:57.870 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:57.870 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T15:59:57.870 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T15:59:57.871 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T15:59:57.871 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:57.871 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:59:57.872 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T15:59:57.872 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T15:59:57.873 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:57.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:57.874 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:57.912 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:57.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:58.175 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:58.175 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:58.175 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:58.176 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:58.176 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:58.176 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:58.176 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.177 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:58.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:58.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:58.177 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:58.178 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:58.178 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665595.365399_75.1", 2020-11-29T15:59:58.178 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:58.178 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:58.178 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:58.179 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.365399Z", 2020-11-29T15:59:58.179 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:58.180 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:58.180 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:58.180 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:58.180 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:58.181 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:58.181 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:58.181 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:58.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:58.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:58.182 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:58.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:58.183 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:58.183 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:58.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:58.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:58.183 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:58.184 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:58.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:58.185 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:58.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:58.186 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:58.186 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:58.186 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:58.186 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:58.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:58.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:58.187 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.188 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:58.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.189 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:58.189 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:58.189 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:58.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.189 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:58.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T15:59:58.232 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-11-29T15:59:58.233 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-7 2020-11-29T15:59:58.263 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-8 2020-11-29T15:59:58.313 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-9 2020-11-29T15:59:58.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:58.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:58.377 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:58.416 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:58.416 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:58.627 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T15:59:58.659 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T15:59:58.659 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:58.660 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665598.354068_89.1", 2020-11-29T15:59:58.660 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:58.354068Z" 2020-11-29T15:59:58.660 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:58.660 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:58.661 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:58.661 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:58.661 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:58.661 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:58.661 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:58.662 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:58.662 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:58.662 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:58.662 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:58.662 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T15:59:58.663 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:58.663 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T15:59:58.664 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T15:59:58.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:58.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:58.665 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:58.707 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:58.707 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:58.960 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:58.961 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:58.961 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:58.961 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:58.962 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:58.962 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:58.962 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.963 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:58.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665598.354068_89.1", 2020-11-29T15:59:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.354068Z", 2020-11-29T15:59:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:58.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:58.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:58.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:58.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:58.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:58.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:58.972 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:58.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:58.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:58.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:58.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:58.973 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:58.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.973 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:58.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:58.974 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:58.974 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:58.974 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:58.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:58.974 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:58.975 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T15:59:58.976 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T15:59:58.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:58.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:58.977 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:59.017 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:59.018 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:59.255 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T15:59:59.255 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T15:59:59.255 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T15:59:59.255 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T15:59:59.256 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T15:59:59.256 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:59.256 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:59.256 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:59.256 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T15:59:59.257 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:59.257 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T15:59:59.257 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:59.257 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:59.257 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665598.354068_89.1", 2020-11-29T15:59:59.258 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:59.258 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T15:59:59.258 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:59.258 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.354068Z", 2020-11-29T15:59:59.259 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:59.259 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:59.259 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:59.259 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:59.259 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T15:59:59.260 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:59.260 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:59.260 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:59.260 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:59.260 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:59.261 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:59.261 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:59.261 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:59.262 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:59.262 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:59.262 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:59.263 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T15:59:59.263 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:59.263 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:59.264 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:59.264 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:59.264 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:59.265 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:59.265 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:59.265 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:59.266 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:59.266 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:59.266 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T15:59:59.267 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T15:59:59.267 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T15:59:59.267 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T15:59:59.268 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T15:59:59.268 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T15:59:59.269 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T15:59:59.269 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T15:59:59.269 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T15:59:59.270 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T15:59:59.270 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:59.270 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:59.271 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T15:59:59.271 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T15:59:59.271 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T15:59:59.272 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T15:59:59.272 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T15:59:59.272 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T15:59:59.273 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T15:59:59.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T15:59:59.274 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T15:59:59.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:59.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:59.276 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:59.316 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:59.317 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T15:59:59.580 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T15:59:59.580 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T15:59:59.580 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T15:59:59.580 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T15:59:59.581 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T15:59:59.581 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T15:59:59.581 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:59.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:59.582 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T15:59:59.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:59.582 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T15:59:59.582 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:59.582 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:59.583 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665598.354068_89.1", 2020-11-29T15:59:59.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:59.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T15:59:59.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:59.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.354068Z", 2020-11-29T15:59:59.584 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:59.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:59.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:59.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:59.585 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T15:59:59.585 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:59.585 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:59.585 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:59.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:59.586 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:59.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:59.586 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:59.586 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:59.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:59.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:59.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:59.587 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T15:59:59.587 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:59.588 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:59.588 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:59.588 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:59.588 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:59.588 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:59.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:59.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T15:59:59.590 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T15:59:59.590 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T15:59:59.590 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T15:59:59.591 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T15:59:59.591 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T15:59:59.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T15:59:59.591 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T15:59:59.591 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T15:59:59.592 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T15:59:59.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:59.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:59.592 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T15:59:59.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T15:59:59.593 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T15:59:59.593 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T15:59:59.593 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T15:59:59.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T15:59:59.594 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T15:59:59.595 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T15:59:59.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:59.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T15:59:59.940 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T15:59:59.974 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T15:59:59.974 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:00.175 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:00:00.208 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:00:00.208 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:00.209 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:00.209 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:00:00.209 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:00.209 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:00.210 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:00.210 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:00:00.210 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:00.210 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:00.210 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:00.211 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:00:00.211 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:00.211 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:00.211 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:00.211 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:00:00.212 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:00.212 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:00:00.213 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:00:00.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:00.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:00.214 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:00.255 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:00.255 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:00.506 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:00:00.506 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:00:00.507 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:00:00.507 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:00:00.507 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:00:00.507 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:00.508 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:00.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:00.508 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:00:00.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:00.508 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:00:00.509 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:00.509 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:00.509 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:00.509 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:00.510 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:00:00.510 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:00.510 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:00.510 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:00.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:00:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:00.512 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:00.512 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:00.512 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:00.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:00.513 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:00.513 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:00.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:00.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:00.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:00.514 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:00:00.514 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:00.514 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:00.514 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:00.514 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:00.515 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:00.515 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:00.515 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:00.515 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:00.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:00.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:00.516 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:00.516 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:00:00.516 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:00.517 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:00.517 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:00.517 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:00.517 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:00.517 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:00.518 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:00.518 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:00.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:00.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:00.519 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:00:00.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:00.519 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:00:00.519 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:00:00.519 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:00:00.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:00.520 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:00:00.521 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:00:00.522 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:00:00.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:00.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:00.524 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:00.565 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:00.565 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:00.813 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:00:00.813 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:00:00.814 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:00:00.814 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:00:00.814 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:00:00.814 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:00.814 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:00.815 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:00.815 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:00:00.815 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:00.815 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:00:00.816 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:00.816 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:00.816 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:00.816 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:00.816 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:00:00.817 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:00.817 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:00.817 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:00.817 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:00.817 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:00.818 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:00.818 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:00:00.818 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:00.818 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:00.819 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:00.819 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:00.819 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:00.819 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:00.820 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:00:00.820 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:00.820 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:00.820 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:00.821 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:00.821 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:00:00.821 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:00.821 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:00.822 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:00.822 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:00.822 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:00.822 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:00.822 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:00:00.823 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:00.823 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:00.823 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:00.823 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:00.824 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:00:00.824 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:00.824 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:00.824 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:00.824 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:00.825 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:00.825 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:00.825 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:00:00.825 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:00.825 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:00.826 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:00.826 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:00:00.826 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:00.826 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:00:00.828 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:00:00.828 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:00:00.828 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:00.829 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:00:00.830 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:00:00.830 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:00:00.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:00.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:00.832 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:00.874 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:00.874 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:01.134 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:00:01.135 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:00:01.135 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:00:01.135 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:00:01.136 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:00:01.136 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:01.136 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:01.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:01.136 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:00:01.137 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:01.137 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:00:01.137 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:01.137 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:01.137 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:01.138 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:01.138 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:00:01.138 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:01.138 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:01.138 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:01.139 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:01.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:01.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:01.139 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:00:01.139 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:01.140 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:01.140 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:01.140 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:01.140 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:01.141 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:01.141 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:01.141 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:01.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:01.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:01.142 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:01.142 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:00:01.142 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:01.142 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:01.142 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:01.143 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:01.143 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:01.143 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:01.143 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:01.143 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:00:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:01.146 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:01.146 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:01.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:01.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:01.146 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:00:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:00:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:00:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:00:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:01.148 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:00:01.148 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:00:01.166 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-11-29T16:00:01.166 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-10 2020-11-29T16:00:01.203 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-11 2020-11-29T16:00:01.290 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-12 2020-11-29T16:00:01.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:01.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:01.955 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:01.997 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:01.997 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:02.286 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:00:02.319 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:00:02.319 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:02.319 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:02.320 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:00:02.320 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:02.320 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:02.320 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665601.335813_45.1", 2020-11-29T16:00:02.320 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:00:01.335813Z" 2020-11-29T16:00:02.321 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:02.321 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:02.321 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:02.321 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:00:02.321 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:02.322 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:02.322 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:02.322 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:00:02.322 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:02.322 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:00:02.323 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:00:02.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:02.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:02.325 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:02.367 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:02.368 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:02.609 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:00:02.609 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:00:02.609 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:00:02.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:00:02.610 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:00:02.610 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:02.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:02.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:02.611 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:00:02.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:02.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:00:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:02.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:00:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:02.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:00:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:02.614 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665601.335813_45.1", 2020-11-29T16:00:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:02.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.335813Z", 2020-11-29T16:00:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:02.616 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:00:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:02.617 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:02.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:00:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:02.619 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:02.620 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:02.620 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:02.620 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:02.620 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:02.620 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:02.621 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:02.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:02.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:02.621 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:00:02.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:02.622 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:00:02.622 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:00:02.622 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:00:02.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:02.622 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:00:02.623 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:00:02.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:00:02.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:02.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:02.625 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:02.664 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:02.664 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:02.906 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:00:02.907 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:00:02.907 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:00:02.907 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:00:02.907 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:00:02.908 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:02.908 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:02.908 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:02.908 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:00:02.908 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:02.909 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:00:02.909 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:02.909 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:02.909 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:02.910 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:02.910 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:00:02.910 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:02.910 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:02.910 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:02.911 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:02.911 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:02.911 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:02.911 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:00:02.912 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:02.912 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:02.912 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665601.335813_45.1", 2020-11-29T16:00:02.912 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:02.912 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:02.913 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:02.913 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.335813Z", 2020-11-29T16:00:02.913 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:02.913 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:02.914 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:02.914 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:02.914 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:00:02.914 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:02.914 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:02.915 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:02.915 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:02.915 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:02.915 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:02.916 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:00:02.916 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:02.916 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:02.916 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:02.916 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:02.917 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:00:02.917 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:02.917 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:02.917 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:02.917 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:02.918 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:02.918 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:02.918 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:00:02.918 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:02.919 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:02.919 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:02.919 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:00:02.919 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:02.919 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:00:02.920 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:00:02.920 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:00:02.920 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:02.920 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:00:02.921 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:00:02.922 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:00:02.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:02.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:02.923 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:02.963 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:02.963 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:03.337 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:00:03.338 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:00:03.338 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:00:03.338 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:00:03.339 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:00:03.339 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:03.339 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:03.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:03.340 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:00:03.340 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:03.340 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:00:03.340 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:03.341 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:03.341 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:03.341 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:03.341 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:00:03.341 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:03.342 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:03.342 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:03.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:03.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:03.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:03.343 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:00:03.343 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:03.343 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:03.344 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665601.335813_45.1", 2020-11-29T16:00:03.344 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:03.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:03.344 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:03.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.335813Z", 2020-11-29T16:00:03.345 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:03.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:03.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:03.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:03.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:00:03.347 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:03.347 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:03.347 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:03.348 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:03.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:03.348 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:03.348 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:03.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:03.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:03.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:03.349 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:03.349 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:00:03.349 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:03.350 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:03.350 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:03.350 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:03.350 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:03.351 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:03.351 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:03.351 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:03.351 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:03.351 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:00:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:00:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:00:03.352 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:00:03.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:03.353 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:00:03.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:00:03.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:03.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:03.355 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:03.442 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:03.443 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:03.671 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:03.676+0000 7fdb24c88080 1 bucket sync caught up with source: 2020-11-29T16:00:03.672 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.4.5, 00000000001.6.5, , ] 2020-11-29T16:00:03.672 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.4.5, 00000000001.6.5, , ] 2020-11-29T16:00:03.672 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:03.676+0000 7fdb24c88080 0 bucket checkpoint complete 2020-11-29T16:00:03.702 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-10 zones={test-zone2, test-zone1} 2020-11-29T16:00:03.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:03.719 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:03.720 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:03.720 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:03.721 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:03.721 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:03.727 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:03.727 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:03.728 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:03.728 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:03.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:03.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:03.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:04.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:04.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:04.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:04.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:04.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:04.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:04.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:04.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:04.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:04.842 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-10 zones={test-zone2, test-zone1} 2020-11-29T16:00:04.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:04.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:04.844 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:04.889 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:04.890 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:05.118 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:05.121+0000 7fad9b438080 1 bucket sync caught up with source: 2020-11-29T16:00:05.118 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.4.5, 00000000001.6.5, , ] 2020-11-29T16:00:05.118 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.4.5, 00000000001.6.5, , ] 2020-11-29T16:00:05.119 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:05.121+0000 7fad9b438080 0 bucket checkpoint complete 2020-11-29T16:00:05.146 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-10 zones={test-zone3, test-zone1} 2020-11-29T16:00:05.157 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:05.162 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:05.163 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:05.163 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:05.163 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:05.164 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:05.168 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:05.169 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:05.169 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:05.170 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:05.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:05.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:05.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:05.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:05.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:05.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:05.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:05.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:05.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:06.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:06.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:06.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:06.323 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-10 zones={test-zone3, test-zone1} 2020-11-29T16:00:06.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:06.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:06.325 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:06.373 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:06.374 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:06.609 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:06.612+0000 7f29fab7c080 1 bucket sync caught up with source: 2020-11-29T16:00:06.609 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.4.5, 00000000001.6.5, , ] 2020-11-29T16:00:06.609 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.6.5, , 00000000001.6.5, , , , 00000000001.4.5, 00000000001.6.5, , ] 2020-11-29T16:00:06.610 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:06.612+0000 7f29fab7c080 0 bucket checkpoint complete 2020-11-29T16:00:06.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:06.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:06.743 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:06.793 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:06.793 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:07.028 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:07.033+0000 7f28eebc6080 1 bucket sync caught up with source: 2020-11-29T16:00:07.028 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.7.5, , 00000000001.6.5, , , , 00000000001.9.5, 00000000001.10.5, , ] 2020-11-29T16:00:07.029 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.7.5, , 00000000001.6.5, , , , 00000000001.9.5, 00000000001.10.5, , ] 2020-11-29T16:00:07.029 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:07.033+0000 7f28eebc6080 0 bucket checkpoint complete 2020-11-29T16:00:07.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-11 zones={test-zone1, test-zone2} 2020-11-29T16:00:07.070 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:07.122 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:07.123 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:07.124 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:07.124 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:07.124 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:07.132 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:07.133 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:07.133 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:07.133 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:07.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:07.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:07.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:07.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:07.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:07.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:07.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:07.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:07.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:08.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:08.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:08.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:08.325 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-11 zones={test-zone1, test-zone2} 2020-11-29T16:00:08.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:08.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:08.327 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:08.376 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:08.376 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:08.617 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:08.621+0000 7f0231ff1080 1 bucket sync caught up with source: 2020-11-29T16:00:08.617 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.7.5, , 00000000001.6.5, , , , 00000000001.9.5, 00000000001.10.5, , ] 2020-11-29T16:00:08.618 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.7.5, , 00000000001.6.5, , , , 00000000001.9.5, 00000000001.10.5, , ] 2020-11-29T16:00:08.618 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:08.621+0000 7f0231ff1080 0 bucket checkpoint complete 2020-11-29T16:00:08.645 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-11 zones={test-zone3, test-zone2} 2020-11-29T16:00:08.656 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:08.663 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:08.663 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:08.664 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:08.664 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:08.664 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:08.714 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:08.715 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:08.716 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:08.716 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:08.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:08.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:08.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:09.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:09.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:09.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:09.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:09.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:09.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:09.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:09.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:09.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:09.824 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-11 zones={test-zone3, test-zone2} 2020-11-29T16:00:09.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:09.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:09.825 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:09.871 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:09.872 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:10.119 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:10.121+0000 7f0730147080 1 bucket sync caught up with source: 2020-11-29T16:00:10.119 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.9.5, , 00000000001.8.5, , , , 00000000001.7.5, 00000000001.11.5, , ] 2020-11-29T16:00:10.119 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.9.5, , 00000000001.8.5, , , , 00000000001.7.5, 00000000001.11.5, , ] 2020-11-29T16:00:10.120 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:10.121+0000 7f0730147080 0 bucket checkpoint complete 2020-11-29T16:00:10.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-12 zones={test-zone1, test-zone3} 2020-11-29T16:00:10.153 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:10.158 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:10.159 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:10.159 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:10.160 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:10.160 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:10.164 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:10.165 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:10.165 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:10.166 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:10.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:10.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:10.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:10.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:10.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:10.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:10.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:10.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:10.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:11.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:11.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:11.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:11.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-12 zones={test-zone1, test-zone3} 2020-11-29T16:00:11.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:11.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:11.275 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:11.324 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:11.324 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:11.574 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:11.580+0000 7f61a1854080 1 bucket sync caught up with source: 2020-11-29T16:00:11.574 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.9.5, , 00000000001.8.5, , , , 00000000001.7.5, 00000000001.11.5, , ] 2020-11-29T16:00:11.575 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.9.5, , 00000000001.8.5, , , , 00000000001.7.5, 00000000001.11.5, , ] 2020-11-29T16:00:11.575 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:11.580+0000 7f61a1854080 0 bucket checkpoint complete 2020-11-29T16:00:11.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-12 zones={test-zone2, test-zone3} 2020-11-29T16:00:11.619 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:11.626 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:11.626 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:11.627 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:11.627 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:11.628 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:11.674 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-11-29T16:00:11.675 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-11-29T16:00:11.675 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-11-29T16:00:11.676 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:11.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:11.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:11.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-11-29T16:00:11.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:12.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:12.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-11-29T16:00:12.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:12.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:12.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-11-29T16:00:12.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:12.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:12.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:12.821 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-12 zones={test-zone2, test-zone3} 2020-11-29T16:00:12.822 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-11-29T16:00:12.822 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-13 2020-11-29T16:00:12.851 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-14 2020-11-29T16:00:12.905 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-15 2020-11-29T16:00:13.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:13.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:13.104 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:13.145 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:13.145 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:13.339 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:00:13.393 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:00:13.393 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:13.393 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:00:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665612.941872_57.1", 2020-11-29T16:00:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:00:12.941872Z" 2020-11-29T16:00:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:00:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:00:13.397 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:13.397 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:00:13.398 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:00:13.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:13.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:13.400 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:13.441 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:13.442 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:13.683 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:00:13.684 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:00:13.684 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:00:13.684 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:00:13.684 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:00:13.685 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:13.685 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:13.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:13.685 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:00:13.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:13.686 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:00:13.686 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:13.686 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:13.686 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:13.687 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:13.687 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:00:13.687 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:13.687 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:13.687 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:13.688 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:00:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665612.941872_57.1", 2020-11-29T16:00:13.689 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.941872Z", 2020-11-29T16:00:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:13.690 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:00:13.691 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:13.692 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:13.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:00:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:13.694 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:13.695 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:00:13.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:00:13.697 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:00:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:00:13.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:13.698 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:00:13.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:00:13.700 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:00:13.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:13.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:13.701 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:13.743 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:13.743 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:13.987 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:00:13.987 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:00:13.988 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:00:13.988 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:00:13.988 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:00:13.989 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:13.989 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:13.989 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:13.990 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:00:13.990 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:13.990 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:00:13.991 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:13.991 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:13.991 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:13.992 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:13.992 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:00:13.992 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:13.993 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:13.993 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:13.993 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:13.994 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:13.995 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:13.995 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:00:13.995 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:13.996 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:13.996 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665612.941872_57.1", 2020-11-29T16:00:13.996 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:13.996 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:13.996 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:13.997 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.941872Z", 2020-11-29T16:00:13.997 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:13.997 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:13.997 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:13.998 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:13.998 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:00:13.998 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:13.998 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:13.998 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:13.999 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:13.999 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:13.999 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:13.999 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:00:13.999 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:14.000 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:14.000 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:14.000 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:00:14.000 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:00:14.000 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:00:14.001 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:00:14.001 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:00:14.002 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:00:14.002 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:00:14.002 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:00:14.003 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:00:14.003 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:00:14.004 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:14.004 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:14.004 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:00:14.005 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:00:14.005 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:00:14.005 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:00:14.005 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:00:14.005 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:00:14.006 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:00:14.007 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:00:14.007 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:00:14.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:14.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:00:14.009 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:00:14.052 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:14.052 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:14.319 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:00:14.319 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:00:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:00:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:00:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:00:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:00:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:00:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:00:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:00:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:00:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:00:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:00:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665612.941872_57.1", 2020-11-29T16:00:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.941872Z", 2020-11-29T16:00:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:00:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:00:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:00:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:00:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:00:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:00:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:00:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:00:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:00:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:00:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:00:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:00:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:00:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:00:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:00:14.333 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:00:14.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:00:14.333 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:00:14.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:00:14.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:14.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:14.336 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:14.424 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:14.424 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:14.666 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:14.672+0000 7f9ec3906080 1 bucket sync caught up with source: 2020-11-29T16:00:14.666 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:14.667 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:14.667 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:14.672+0000 7f9ec3906080 0 bucket checkpoint complete 2020-11-29T16:00:14.698 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-13 zones={test-zone2, test-zone1} 2020-11-29T16:00:14.709 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:14.758 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:14.759 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:14.767 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:14.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:14.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:15.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:15.110 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-13 zones={test-zone2, test-zone1} 2020-11-29T16:00:15.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:15.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:15.112 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:15.158 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:15.158 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:15.364 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:15.369+0000 7f9d4e085080 1 bucket sync caught up with source: 2020-11-29T16:00:15.365 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:15.365 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:15.365 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:15.369+0000 7f9d4e085080 0 bucket checkpoint complete 2020-11-29T16:00:15.388 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-13 zones={test-zone3, test-zone1} 2020-11-29T16:00:15.396 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:15.446 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:15.447 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:15.455 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:15.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:15.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:15.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:15.789 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-13 zones={test-zone3, test-zone1} 2020-11-29T16:00:15.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:15.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:15.791 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:15.839 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:15.839 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:16.064 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:16.068+0000 7ff8f3293080 1 bucket sync caught up with source: 2020-11-29T16:00:16.065 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:16.065 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:16.065 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:16.068+0000 7ff8f3293080 0 bucket checkpoint complete 2020-11-29T16:00:16.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:16.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:16.100 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:16.149 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:16.149 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:16.389 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:16.393+0000 7f8fd10ff080 1 bucket sync caught up with source: 2020-11-29T16:00:16.390 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-29T16:00:16.390 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2020-11-29T16:00:16.390 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:16.393+0000 7f8fd10ff080 0 bucket checkpoint complete 2020-11-29T16:00:16.420 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-14 zones={test-zone1, test-zone2} 2020-11-29T16:00:16.432 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:16.482 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:16.483 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:16.490 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:16.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:16.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:16.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:16.817 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-14 zones={test-zone1, test-zone2} 2020-11-29T16:00:16.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:16.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:16.818 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:16.866 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:16.866 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:17.114 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:17.117+0000 7f74f714b080 1 bucket sync caught up with source: 2020-11-29T16:00:17.114 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-29T16:00:17.115 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.6.5, , , , , , , , , ] 2020-11-29T16:00:17.115 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:17.117+0000 7f74f714b080 0 bucket checkpoint complete 2020-11-29T16:00:17.144 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-14 zones={test-zone3, test-zone2} 2020-11-29T16:00:17.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:17.206 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:17.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:17.215 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:17.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:17.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:17.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:17.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-14 zones={test-zone3, test-zone2} 2020-11-29T16:00:17.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:17.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:17.539 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:17.587 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:17.587 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:17.826 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:17.829+0000 7fe9fa258080 1 bucket sync caught up with source: 2020-11-29T16:00:17.827 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.13.5, , , , , , , , , ] 2020-11-29T16:00:17.827 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.13.5, , , , , , , , , ] 2020-11-29T16:00:17.827 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:17.829+0000 7fe9fa258080 0 bucket checkpoint complete 2020-11-29T16:00:17.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-15 zones={test-zone1, test-zone3} 2020-11-29T16:00:17.862 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:17.910 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:17.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:17.919 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:17.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:18.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:18.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:18.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-15 zones={test-zone1, test-zone3} 2020-11-29T16:00:18.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:18.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:18.247 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:18.294 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:18.295 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:18.526 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:18.532+0000 7fc9c2de0080 1 bucket sync caught up with source: 2020-11-29T16:00:18.526 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.13.5, , , , , , , , , ] 2020-11-29T16:00:18.527 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.13.5, , , , , , , , , ] 2020-11-29T16:00:18.527 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:18.532+0000 7fc9c2de0080 0 bucket checkpoint complete 2020-11-29T16:00:18.557 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-15 zones={test-zone2, test-zone3} 2020-11-29T16:00:18.569 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:18.618 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:18.619 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:18.627 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-11-29T16:00:18.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:18.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:18.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-11-29T16:00:18.958 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-15 zones={test-zone2, test-zone3} 2020-11-29T16:00:18.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:18.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:18.976 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:19.022 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:19.022 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:19.259 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:19.264+0000 7f85a0a8f080 1 waiting for incremental sync to catch up: 2020-11-29T16:00:19.259 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:19.259 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:24.262 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:24.268+0000 7f85a0a8f080 1 waiting for incremental sync to catch up: 2020-11-29T16:00:24.263 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:24.263 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:29.266 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:29.272+0000 7f85a0a8f080 1 waiting for incremental sync to catch up: 2020-11-29T16:00:29.267 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:29.267 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:34.270 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:34.276+0000 7f85a0a8f080 1 waiting for incremental sync to catch up: 2020-11-29T16:00:34.270 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-11-29T16:00:34.270 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:39.274 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:39.280+0000 7f85a0a8f080 1 bucket sync caught up with source: 2020-11-29T16:00:39.274 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:39.274 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:39.275 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:39.280+0000 7f85a0a8f080 0 bucket checkpoint complete 2020-11-29T16:00:39.307 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-13 zones={test-zone2, test-zone1} 2020-11-29T16:00:39.319 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:39.370 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:39.439 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-13 zones={test-zone2, test-zone1} 2020-11-29T16:00:39.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:39.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:39.441 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:39.486 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:39.487 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:39.720 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:39.725+0000 7f626895a080 1 bucket sync caught up with source: 2020-11-29T16:00:39.721 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:39.721 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:39.721 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:39.725+0000 7f626895a080 0 bucket checkpoint complete 2020-11-29T16:00:39.752 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-13 zones={test-zone3, test-zone1} 2020-11-29T16:00:39.759 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:00:39.806 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:00:39.864 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-13 zones={test-zone3, test-zone1} 2020-11-29T16:00:39.865 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:39.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:39.866 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:39.908 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:39.909 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:40.154 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:40.160+0000 7f3a349cd080 1 bucket sync caught up with source: 2020-11-29T16:00:40.154 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:40.155 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-11-29T16:00:40.155 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:00:40.160+0000 7f3a349cd080 0 bucket checkpoint complete 2020-11-29T16:00:40.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:40.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:00:40.295 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:00:40.341 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:00:40.341 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:00:40.556 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:40.561+0000 7fcaa9be5080 1 waiting for incremental sync to catch up: 2020-11-29T16:00:40.556 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-29T16:00:40.556 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000002.9.5, , , , , , , , , ] 2020-11-29T16:00:45.559 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:45.565+0000 7fcaa9be5080 1 waiting for incremental sync to catch up: 2020-11-29T16:00:45.560 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-29T16:00:45.560 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000002.9.5, , , , , , , , , ] 2020-11-29T16:00:50.564 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:50.569+0000 7fcaa9be5080 1 waiting for incremental sync to catch up: 2020-11-29T16:00:50.564 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-29T16:00:50.564 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000002.9.5, , , , , , , , , ] 2020-11-29T16:00:55.568 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:00:55.573+0000 7fcaa9be5080 1 waiting for incremental sync to catch up: 2020-11-29T16:00:55.568 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.6.5, , , , , , , , , ] 2020-11-29T16:00:55.568 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000002.9.5, , , , , , , , , ] 2020-11-29T16:01:00.571 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:00.577+0000 7fcaa9be5080 1 bucket sync caught up with source: 2020-11-29T16:01:00.572 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000002.9.5, , , , , , , , , ] 2020-11-29T16:01:00.572 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000002.9.5, , , , , , , , , ] 2020-11-29T16:01:00.572 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:00.577+0000 7fcaa9be5080 0 bucket checkpoint complete 2020-11-29T16:01:00.601 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-14 zones={test-zone1, test-zone2} 2020-11-29T16:01:00.612 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:01:00.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:01:00.729 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-14 zones={test-zone1, test-zone2} 2020-11-29T16:01:00.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:00.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:00.731 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:01:00.777 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:00.778 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:01.010 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:01.013+0000 7fab5cbce080 1 bucket sync caught up with source: 2020-11-29T16:01:01.010 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000002.9.5, , , , , , , , , ] 2020-11-29T16:01:01.011 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000002.9.5, , , , , , , , , ] 2020-11-29T16:01:01.011 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:01.013+0000 7fab5cbce080 0 bucket checkpoint complete 2020-11-29T16:01:01.041 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-14 zones={test-zone3, test-zone2} 2020-11-29T16:01:01.051 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:01:01.102 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:01:01.165 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-14 zones={test-zone3, test-zone2} 2020-11-29T16:01:01.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:01.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:01.191 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:01:01.235 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:01.235 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:01.454 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:01.461+0000 7effa7572080 1 bucket sync caught up with source: 2020-11-29T16:01:01.455 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000002.16.5, , , , , , , , , ] 2020-11-29T16:01:01.455 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000002.16.5, , , , , , , , , ] 2020-11-29T16:01:01.455 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:01.461+0000 7effa7572080 0 bucket checkpoint complete 2020-11-29T16:01:01.486 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-15 zones={test-zone1, test-zone3} 2020-11-29T16:01:01.493 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:01:01.538 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:01:01.600 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-15 zones={test-zone1, test-zone3} 2020-11-29T16:01:01.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:01.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:01.604 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:01:01.652 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:01.652 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:01.881 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:01:01.888+0000 7f16fd1bf080 1 bucket sync caught up with source: 2020-11-29T16:01:01.881 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.16.5, , , , , , , , , ] 2020-11-29T16:01:01.882 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.16.5, , , , , , , , , ] 2020-11-29T16:01:01.882 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:01:01.888+0000 7f16fd1bf080 0 bucket checkpoint complete 2020-11-29T16:01:01.911 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-15 zones={test-zone2, test-zone3} 2020-11-29T16:01:01.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:01:01.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:01:02.033 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-15 zones={test-zone2, test-zone3} 2020-11-29T16:01:02.033 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-11-29T16:01:02.034 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-16 2020-11-29T16:01:02.068 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-17 2020-11-29T16:01:02.126 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-18 2020-11-29T16:01:02.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:02.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:02.236 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:01:02.274 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:02.274 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:02.483 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:01:02.514 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:01:02.515 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:01:02.515 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:01:02.515 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:01:02.515 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:01:02.515 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:01:02.516 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665662.229493_76.1", 2020-11-29T16:01:02.516 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:01:02.229493Z" 2020-11-29T16:01:02.516 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:01:02.516 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:01:02.517 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:01:02.517 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:01:02.517 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:01:02.517 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:01:02.517 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:01:02.518 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:01:02.518 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:01:02.518 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:01:02.519 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:01:02.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:02.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:02.521 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:01:02.561 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:02.562 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:02.821 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:01:02.821 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:01:02.821 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:01:02.821 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:01:02.822 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:01:02.822 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:01:02.822 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:02.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:02.823 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:01:02.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:02.823 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:01:02.823 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:02.824 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:02.824 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:01:02.824 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:02.824 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:01:02.824 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:02.825 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:01:02.825 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:02.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:02.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:02.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:02.826 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:01:02.827 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:02.827 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:02.827 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665662.195335_72.1", 2020-11-29T16:01:02.828 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:02.828 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:02.828 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:02.828 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.195335Z", 2020-11-29T16:01:02.828 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:02.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:02.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:02.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:02.829 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:01:02.830 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:02.830 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:02.830 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:01:02.830 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:02.831 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:02.831 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:02.831 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:01:02.831 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:02.831 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:02.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:02.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:02.832 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:01:02.832 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:02.832 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:02.833 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:01:02.833 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:02.833 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:02.833 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:02.833 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:01:02.834 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:02.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:02.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:02.834 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:01:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:01:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:01:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:01:02.835 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:02.836 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:01:02.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606665662.229493_76.1 target=1_1606665662.195335_72.1 2020-11-29T16:01:07.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:07.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:07.839 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:01:07.881 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:07.882 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:08.139 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:01:08.139 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:01:08.139 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:01:08.139 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:01:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:01:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:01:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:08.140 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:01:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:01:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:08.141 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:01:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:01:08.142 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:01:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:08.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:01:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:08.144 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665662.195335_72.1", 2020-11-29T16:01:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:08.145 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.195335Z", 2020-11-29T16:01:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:08.146 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:01:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:08.147 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:01:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:01:08.148 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:08.149 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:08.150 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:01:08.150 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:08.150 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:08.151 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:01:08.151 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:08.152 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:08.152 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:08.152 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:01:08.153 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:08.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:08.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:08.154 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:01:08.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:08.155 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:01:08.155 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:01:08.155 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:01:08.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:08.156 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:01:08.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606665662.229493_76.1 target=1_1606665662.195335_72.1 2020-11-29T16:01:13.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:13.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:13.160 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:01:13.201 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:13.201 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:13.459 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:01:13.459 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:01:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:01:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:01:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:01:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:01:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:01:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:01:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:01:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:01:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:01:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:13.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:13.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:13.464 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:01:13.464 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:13.464 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665662.229493_76.1", 2020-11-29T16:01:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.229493Z", 2020-11-29T16:01:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:01:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:01:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:01:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:01:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:01:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:01:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:01:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:01:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:01:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:01:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:13.473 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:01:13.474 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:01:13.474 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:01:13.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:13.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:13.476 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:01:13.517 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:13.517 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:13.774 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:01:13.774 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:01:13.775 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:01:13.775 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:01:13.775 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:01:13.775 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:01:13.775 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:01:13.776 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:01:13.776 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:01:13.776 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:01:13.776 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:01:13.777 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:01:13.777 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:01:13.777 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:01:13.777 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:01:13.778 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:01:13.778 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:01:13.778 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:01:13.778 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:01:13.778 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:01:13.779 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:01:13.779 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:01:13.779 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:01:13.779 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:01:13.780 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:01:13.780 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665662.229493_76.1", 2020-11-29T16:01:13.780 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:01:13.780 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:01:13.780 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:01:13.781 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.229493Z", 2020-11-29T16:01:13.781 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:01:13.781 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:01:13.781 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:01:13.781 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:01:13.782 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:01:13.782 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:01:13.782 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:01:13.782 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:01:13.782 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:01:13.783 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:01:13.783 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:01:13.783 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:01:13.783 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:01:13.783 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:01:13.784 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:01:13.784 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:01:13.784 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:01:13.784 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:01:13.784 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:01:13.785 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:01:13.785 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:01:13.785 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:01:13.786 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:01:13.786 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:01:13.786 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:01:13.787 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:01:13.787 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:01:13.787 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:01:13.788 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:01:13.788 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:01:13.788 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:01:13.788 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:01:13.789 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:01:13.789 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:01:13.790 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:01:13.790 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:01:13.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:13.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:01:13.792 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:01:13.825 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:13.826 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:14.075 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:01:14.075 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:01:14.076 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:01:14.077 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:01:14.077 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:01:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:01:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:01:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:01:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:01:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:01:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:01:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:01:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665662.229493_76.1", 2020-11-29T16:01:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.229493Z", 2020-11-29T16:01:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:01:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:01:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:01:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:01:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:01:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:01:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:01:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:01:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:01:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:01:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:01:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:01:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:01:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:01:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:01:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:01:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:01:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:01:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:01:14.090 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:01:14.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:01:14.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:14.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:14.144 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:01:14.193 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:14.193 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:14.434 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:01:14.440+0000 7fb19e63a080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:14.434 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:01:14.435 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2020-11-29T16:01:19.437 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:01:19.444+0000 7fb19e63a080 1 bucket sync caught up with source: 2020-11-29T16:01:19.438 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:19.438 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.7.10, , , , , ] 2020-11-29T16:01:19.438 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:01:19.444+0000 7fb19e63a080 0 bucket checkpoint complete 2020-11-29T16:01:19.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:19.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:01:19.476 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:01:19.522 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:01:19.522 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:01:19.752 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:19.757+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:19.752 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:19.753 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:01:24.755 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:24.761+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:24.756 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:24.756 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:01:29.761 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:29.765+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:29.762 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:29.762 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:01:34.763 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:34.769+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:34.764 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:34.764 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:01:39.766 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:39.769+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:39.766 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:39.766 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:01:44.770 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:44.777+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:44.771 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:44.771 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:01:49.774 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:49.781+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:49.774 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:49.775 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:01:54.778 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:54.785+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:54.779 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:54.779 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:01:59.782 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:01:59.789+0000 7f405ce6c080 1 waiting for incremental sync to catch up: 2020-11-29T16:01:59.783 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.11.9, , , , , ] 2020-11-29T16:01:59.783 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:02:04.786 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:04.792+0000 7f405ce6c080 1 bucket sync caught up with source: 2020-11-29T16:02:04.787 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:02:04.787 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:02:04.787 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:04.792+0000 7f405ce6c080 0 bucket checkpoint complete 2020-11-29T16:02:04.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:04.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:04.820 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:04.871 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:04.872 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:05.108 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:05.115+0000 7fe670e0a080 1 bucket sync caught up with source: 2020-11-29T16:02:05.109 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:02:05.109 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.13.9, , , , , ] 2020-11-29T16:02:05.109 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:05.115+0000 7fe670e0a080 0 bucket checkpoint complete 2020-11-29T16:02:05.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:05.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:05.284 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:05.331 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:05.332 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:05.546 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:05.552+0000 7fa788368080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:05.547 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:02:05.547 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:10.551 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:10.555+0000 7fa788368080 1 bucket sync caught up with source: 2020-11-29T16:02:10.551 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:10.552 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:10.552 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:10.555+0000 7fa788368080 0 bucket checkpoint complete 2020-11-29T16:02:10.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:10.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:10.590 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:10.637 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:10.638 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:10.884 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:10.887+0000 7fa21737f080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:10.884 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:10.885 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000006.29.9, , , , , ] 2020-11-29T16:02:15.888 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:15.890+0000 7fa21737f080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:15.889 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:15.889 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000006.29.9, , , , , ] 2020-11-29T16:02:20.893 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:20.894+0000 7fa21737f080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:20.893 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:20.894 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000006.29.9, , , , , ] 2020-11-29T16:02:25.898 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:25.901+0000 7fa21737f080 1 bucket sync caught up with source: 2020-11-29T16:02:25.898 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000006.29.9, , , , , ] 2020-11-29T16:02:25.899 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000006.29.9, , , , , ] 2020-11-29T16:02:25.899 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:25.901+0000 7fa21737f080 0 bucket checkpoint complete 2020-11-29T16:02:25.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:25.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:25.988 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:26.035 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:26.035 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:26.258 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:26.261+0000 7f687d6db080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:26.258 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:02:26.259 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:31.263 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:31.264+0000 7f687d6db080 1 bucket sync caught up with source: 2020-11-29T16:02:31.263 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:31.263 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:31.264 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:31.264+0000 7f687d6db080 0 bucket checkpoint complete 2020-11-29T16:02:31.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:31.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:31.295 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:31.341 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:31.341 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:31.576 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:31.574+0000 7efcaf612080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:31.576 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:31.576 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.25.9, , , , , ] 2020-11-29T16:02:36.581 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:36.582+0000 7efcaf612080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:36.582 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:36.582 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.25.9, , , , , ] 2020-11-29T16:02:41.586 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:41.585+0000 7efcaf612080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:41.586 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:41.587 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.25.9, , , , , ] 2020-11-29T16:02:46.592 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:46.592+0000 7efcaf612080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:46.592 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:46.592 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.25.9, , , , , ] 2020-11-29T16:02:51.596 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:51.596+0000 7efcaf612080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:51.596 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.21.10, , , , , ] 2020-11-29T16:02:51.597 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.25.9, , , , , ] 2020-11-29T16:02:56.601 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:56.600+0000 7efcaf612080 1 bucket sync caught up with source: 2020-11-29T16:02:56.601 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.25.9, , , , , ] 2020-11-29T16:02:56.602 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.25.9, , , , , ] 2020-11-29T16:02:56.602 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:56.600+0000 7efcaf612080 0 bucket checkpoint complete 2020-11-29T16:02:58.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:58.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:58.355 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:58.403 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:58.403 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:58.632 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:58.631+0000 7fac978b1080 1 bucket sync caught up with source: 2020-11-29T16:02:58.632 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.26.5, , 00000000003.19.9, , 00000000008.25.9, , , , , ] 2020-11-29T16:02:58.633 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.26.5, , 00000000003.19.9, , 00000000008.25.9, , , , , ] 2020-11-29T16:02:58.633 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:58.631+0000 7fac978b1080 0 bucket checkpoint complete 2020-11-29T16:02:58.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:58.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:58.667 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:58.715 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:58.716 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:58.951 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:58.946+0000 7fe0f7da6080 1 bucket sync caught up with source: 2020-11-29T16:02:58.951 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000010.26.5, , 00000000003.19.9, , 00000000008.25.9, , , , , ] 2020-11-29T16:02:58.951 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000010.26.5, , 00000000003.19.9, , 00000000008.25.9, , , , , ] 2020-11-29T16:02:58.951 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:58.946+0000 7fe0f7da6080 0 bucket checkpoint complete 2020-11-29T16:02:58.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:58.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:58.978 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:59.026 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:59.026 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:59.273 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:59.271+0000 7f05ffe70080 1 bucket sync caught up with source: 2020-11-29T16:02:59.273 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.26.5, , 00000000003.19.9, , 00000000008.25.9, , , , , ] 2020-11-29T16:02:59.274 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.26.5, , 00000000003.19.9, , 00000000008.25.9, , , , , ] 2020-11-29T16:02:59.274 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:02:59.271+0000 7f05ffe70080 0 bucket checkpoint complete 2020-11-29T16:02:59.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:59.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:02:59.398 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:02:59.448 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:02:59.449 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:02:59.688 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:02:59.686+0000 7fc1ad236080 1 waiting for incremental sync to catch up: 2020-11-29T16:02:59.689 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.15.9, , 00000000010.35.5, , 00000000006.39.10, , , , , ] 2020-11-29T16:02:59.689 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.15.9, , 00000000010.35.5, , 00000000007.44.9, , , , , ] 2020-11-29T16:03:04.693 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:04.690+0000 7fc1ad236080 1 waiting for incremental sync to catch up: 2020-11-29T16:03:04.693 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.15.9, , 00000000010.35.5, , 00000000006.39.10, , , , , ] 2020-11-29T16:03:04.693 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.15.9, , 00000000010.35.5, , 00000000007.44.9, , , , , ] 2020-11-29T16:03:09.696 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:09.694+0000 7fc1ad236080 1 waiting for incremental sync to catch up: 2020-11-29T16:03:09.697 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.15.9, , 00000000010.35.5, , 00000000006.39.10, , , , , ] 2020-11-29T16:03:09.697 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.15.9, , 00000000010.35.5, , 00000000007.44.9, , , , , ] 2020-11-29T16:03:14.701 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:14.697+0000 7fc1ad236080 1 bucket sync caught up with source: 2020-11-29T16:03:14.701 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.15.9, , 00000000010.35.5, , 00000000007.44.9, , , , , ] 2020-11-29T16:03:14.702 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.15.9, , 00000000010.35.5, , 00000000007.44.9, , , , , ] 2020-11-29T16:03:14.702 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:14.697+0000 7fc1ad236080 0 bucket checkpoint complete 2020-11-29T16:03:14.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:14.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:14.736 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:14.785 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:14.786 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:15.015 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:15.009+0000 7fb28e8c9080 1 bucket sync caught up with source: 2020-11-29T16:03:15.016 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.15.9, , 00000000010.35.5, , 00000000007.44.9, , , , , ] 2020-11-29T16:03:15.016 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.15.9, , 00000000010.35.5, , 00000000007.44.9, , , , , ] 2020-11-29T16:03:15.016 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:15.009+0000 7fb28e8c9080 0 bucket checkpoint complete 2020-11-29T16:03:15.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:15.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:15.049 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:15.095 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:15.096 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:15.331 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:15.325+0000 7faeb5265080 1 waiting for incremental sync to catch up: 2020-11-29T16:03:15.331 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.17.9, , 00000000002.19.10, , 00000000014.39.5, , , , , ] 2020-11-29T16:03:15.332 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.17.9, , 00000000003.41.9, , 00000000014.39.5, , , , , ] 2020-11-29T16:03:20.334 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:20.329+0000 7faeb5265080 1 bucket sync caught up with source: 2020-11-29T16:03:20.335 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.17.9, , 00000000003.41.9, , 00000000014.39.5, , , , , ] 2020-11-29T16:03:20.335 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.17.9, , 00000000003.41.9, , 00000000014.39.5, , , , , ] 2020-11-29T16:03:20.335 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:20.329+0000 7faeb5265080 0 bucket checkpoint complete 2020-11-29T16:03:20.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:20.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:20.364 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:20.413 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:20.652 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:20.650+0000 7fe974f06080 1 bucket sync caught up with source: 2020-11-29T16:03:20.652 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.17.9, , 00000000003.41.9, , 00000000014.39.5, , , , , ] 2020-11-29T16:03:20.652 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.17.9, , 00000000003.41.9, , 00000000014.39.5, , , , , ] 2020-11-29T16:03:20.653 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:20.650+0000 7fe974f06080 0 bucket checkpoint complete 2020-11-29T16:03:20.681 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-16 zones={test-zone2, test-zone1} 2020-11-29T16:03:20.691 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:20.697 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:20.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:20.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:20.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:20.699 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:20.706 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:20.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:20.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:20.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:20.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:20.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:20.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:20.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:20.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:21.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:21.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:21.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:21.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:21.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:21.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:21.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:21.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-16 zones={test-zone2, test-zone1} 2020-11-29T16:03:21.710 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-16 zones={test-zone3, test-zone1} 2020-11-29T16:03:21.719 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:21.723 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:21.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:21.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:21.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:21.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:21.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:21.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:21.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:21.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:21.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:21.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:21.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:21.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:21.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:22.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:22.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:22.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:22.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:22.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:22.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:22.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:22.729 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-16 zones={test-zone3, test-zone1} 2020-11-29T16:03:22.730 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-16 zones={test-zone3, test-zone2} 2020-11-29T16:03:22.741 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:22.746 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:22.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:22.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:22.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:22.748 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:22.755 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:22.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:22.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:22.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:22.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:22.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:22.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:22.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:22.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:23.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:23.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:23.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:23.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:23.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:23.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:23.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:23.757 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-16 zones={test-zone3, test-zone2} 2020-11-29T16:03:23.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:23.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:23.759 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:23.803 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:23.803 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:24.002 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:23.997+0000 7f7550a43080 1 waiting for incremental sync to catch up: 2020-11-29T16:03:24.003 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.25.5, , 00000000003.12.9, , 00000000004.20.9, , , , , ] 2020-11-29T16:03:24.003 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.25.5, , 00000000003.12.9, , 00000000007.26.9, , , , , ] 2020-11-29T16:03:29.006 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:29.001+0000 7f7550a43080 1 waiting for incremental sync to catch up: 2020-11-29T16:03:29.006 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.25.5, , 00000000003.12.9, , 00000000004.20.9, , , , , ] 2020-11-29T16:03:29.006 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.25.5, , 00000000003.12.9, , 00000000007.26.9, , , , , ] 2020-11-29T16:03:34.010 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:34.005+0000 7f7550a43080 1 bucket sync caught up with source: 2020-11-29T16:03:34.011 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.25.5, , 00000000003.12.9, , 00000000007.26.9, , , , , ] 2020-11-29T16:03:34.011 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.25.5, , 00000000003.12.9, , 00000000007.26.9, , , , , ] 2020-11-29T16:03:34.011 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:34.005+0000 7f7550a43080 0 bucket checkpoint complete 2020-11-29T16:03:34.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:34.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:34.043 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:34.091 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:34.092 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:34.306 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:34.305+0000 7fde6f8e1080 1 bucket sync caught up with source: 2020-11-29T16:03:34.306 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000010.25.5, , 00000000003.12.9, , 00000000007.26.9, , , , , ] 2020-11-29T16:03:34.306 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000010.25.5, , 00000000003.12.9, , 00000000007.26.9, , , , , ] 2020-11-29T16:03:34.306 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:34.305+0000 7fde6f8e1080 0 bucket checkpoint complete 2020-11-29T16:03:34.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:34.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:34.331 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:34.377 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:34.378 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:34.614 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:34.608+0000 7fe21d700080 1 bucket sync caught up with source: 2020-11-29T16:03:34.614 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.25.5, , 00000000003.12.9, , 00000000007.26.9, , , , , ] 2020-11-29T16:03:34.614 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.25.5, , 00000000003.12.9, , 00000000007.26.9, , , , , ] 2020-11-29T16:03:34.614 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:34.608+0000 7fe21d700080 0 bucket checkpoint complete 2020-11-29T16:03:34.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:34.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:34.746 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:34.794 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:34.795 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:35.040 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:35.041+0000 7fc88d96d080 1 bucket sync caught up with source: 2020-11-29T16:03:35.041 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.16.9, , 00000000010.29.5, , 00000000009.35.9, , , , , ] 2020-11-29T16:03:35.041 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.16.9, , 00000000010.29.5, , 00000000009.35.9, , , , , ] 2020-11-29T16:03:35.041 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:35.041+0000 7fc88d96d080 0 bucket checkpoint complete 2020-11-29T16:03:35.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:35.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:35.073 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:35.120 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:35.121 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:35.376 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:35.373+0000 7f4cf828f080 1 bucket sync caught up with source: 2020-11-29T16:03:35.377 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.16.9, , 00000000010.29.5, , 00000000009.35.9, , , , , ] 2020-11-29T16:03:35.377 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.16.9, , 00000000010.29.5, , 00000000009.35.9, , , , , ] 2020-11-29T16:03:35.378 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:35.373+0000 7f4cf828f080 0 bucket checkpoint complete 2020-11-29T16:03:35.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:35.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:35.411 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:35.501 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:35.501 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:35.736 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:35.733+0000 7f9a6c09b080 1 bucket sync caught up with source: 2020-11-29T16:03:35.737 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.14.9, , 00000000005.19.5, , 00000000014.43.5, , , , , ] 2020-11-29T16:03:35.737 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.14.9, , 00000000005.19.5, , 00000000014.43.5, , , , , ] 2020-11-29T16:03:35.737 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:35.733+0000 7f9a6c09b080 0 bucket checkpoint complete 2020-11-29T16:03:35.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:35.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:35.766 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:35.822 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:35.823 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:36.042 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:36.040+0000 7f84de292080 1 bucket sync caught up with source: 2020-11-29T16:03:36.042 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.14.9, , 00000000005.19.5, , 00000000014.43.5, , , , , ] 2020-11-29T16:03:36.042 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.14.9, , 00000000005.19.5, , 00000000014.43.5, , , , , ] 2020-11-29T16:03:36.042 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:36.040+0000 7f84de292080 0 bucket checkpoint complete 2020-11-29T16:03:36.074 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-17 zones={test-zone2, test-zone1} 2020-11-29T16:03:36.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:36.094 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:36.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:36.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:36.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:36.096 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:36.103 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:36.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:36.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:36.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:36.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:36.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:36.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:36.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:36.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:36.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:36.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:36.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:36.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:36.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:36.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:37.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:37.114 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-17 zones={test-zone2, test-zone1} 2020-11-29T16:03:37.115 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-17 zones={test-zone3, test-zone1} 2020-11-29T16:03:37.125 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:37.130 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:37.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:37.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:37.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:37.132 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:37.137 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:37.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:37.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:37.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:37.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:37.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:37.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:37.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:37.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:37.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:37.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:37.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:37.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:37.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:37.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:38.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:38.170 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-17 zones={test-zone3, test-zone1} 2020-11-29T16:03:38.171 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-17 zones={test-zone3, test-zone2} 2020-11-29T16:03:38.182 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:38.188 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:38.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:38.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:38.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:38.189 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:38.238 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:38.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:38.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:38.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:38.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:38.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:38.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:38.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:38.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:38.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:38.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:38.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:38.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:38.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:39.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:39.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:39.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-17 zones={test-zone3, test-zone2} 2020-11-29T16:03:39.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:39.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:39.245 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:39.292 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:39.292 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:39.526 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:39.524+0000 7f26a7bc8080 1 bucket sync caught up with source: 2020-11-29T16:03:39.526 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.25.5, , 00000000003.22.9, , 00000000007.28.9, , , , , ] 2020-11-29T16:03:39.526 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.25.5, , 00000000003.22.9, , 00000000007.28.9, , , , , ] 2020-11-29T16:03:39.527 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:39.524+0000 7f26a7bc8080 0 bucket checkpoint complete 2020-11-29T16:03:39.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:39.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:39.560 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:39.611 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:39.612 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:39.824 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:39.821+0000 7f04f28b2080 1 bucket sync caught up with source: 2020-11-29T16:03:39.824 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000010.25.5, , 00000000003.22.9, , 00000000007.28.9, , , , , ] 2020-11-29T16:03:39.825 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000010.25.5, , 00000000003.22.9, , 00000000007.28.9, , , , , ] 2020-11-29T16:03:39.825 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:39.821+0000 7f04f28b2080 0 bucket checkpoint complete 2020-11-29T16:03:39.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:39.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:39.857 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:39.904 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:39.905 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:40.130 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:40.128+0000 7fe608480080 1 bucket sync caught up with source: 2020-11-29T16:03:40.131 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.25.5, , 00000000003.22.9, , 00000000007.28.9, , , , , ] 2020-11-29T16:03:40.131 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.25.5, , 00000000003.22.9, , 00000000007.28.9, , , , , ] 2020-11-29T16:03:40.131 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:40.128+0000 7fe608480080 0 bucket checkpoint complete 2020-11-29T16:03:40.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:40.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:40.168 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:40.217 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:40.217 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:40.459 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:40.457+0000 7f7c9496a080 1 bucket sync caught up with source: 2020-11-29T16:03:40.460 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.14.9, , 00000000010.41.5, , 00000000007.50.9, , , , , ] 2020-11-29T16:03:40.460 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.14.9, , 00000000010.41.5, , 00000000007.50.9, , , , , ] 2020-11-29T16:03:40.460 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:40.457+0000 7f7c9496a080 0 bucket checkpoint complete 2020-11-29T16:03:40.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:40.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:40.501 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:40.549 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:40.549 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:40.791 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:40.789+0000 7f4a15f42080 1 bucket sync caught up with source: 2020-11-29T16:03:40.791 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.14.9, , 00000000010.41.5, , 00000000007.50.9, , , , , ] 2020-11-29T16:03:40.791 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.14.9, , 00000000010.41.5, , 00000000007.50.9, , , , , ] 2020-11-29T16:03:40.792 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:40.789+0000 7f4a15f42080 0 bucket checkpoint complete 2020-11-29T16:03:40.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:40.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:40.820 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:40.868 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:40.868 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:41.113 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:41.113+0000 7fdd8faca080 1 bucket sync caught up with source: 2020-11-29T16:03:41.113 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000003.15.9, , 00000000003.53.9, , 00000000015.51.5, , , , , ] 2020-11-29T16:03:41.114 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000003.15.9, , 00000000003.53.9, , 00000000015.51.5, , , , , ] 2020-11-29T16:03:41.114 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:41.113+0000 7fdd8faca080 0 bucket checkpoint complete 2020-11-29T16:03:41.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:41.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:41.142 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:41.189 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:41.189 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:41.437 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:41.436+0000 7f7e3e61b080 1 bucket sync caught up with source: 2020-11-29T16:03:41.437 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.15.9, , 00000000003.53.9, , 00000000015.51.5, , , , , ] 2020-11-29T16:03:41.437 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.15.9, , 00000000003.53.9, , 00000000015.51.5, , , , , ] 2020-11-29T16:03:41.438 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:41.436+0000 7f7e3e61b080 0 bucket checkpoint complete 2020-11-29T16:03:41.468 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-18 zones={test-zone2, test-zone1} 2020-11-29T16:03:41.478 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:41.485 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:41.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:41.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:41.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:41.487 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:41.493 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:41.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:41.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:41.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:41.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:41.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:41.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:41.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:41.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:41.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:41.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:41.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:42.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:42.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:42.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:42.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:42.502 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-18 zones={test-zone2, test-zone1} 2020-11-29T16:03:42.503 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-18 zones={test-zone3, test-zone1} 2020-11-29T16:03:42.511 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:42.558 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:42.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:42.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:42.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:42.560 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:42.569 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:42.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:42.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:42.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:42.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:42.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:42.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:42.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:42.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:42.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:43.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:43.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:43.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:43.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:43.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:43.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:43.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-18 zones={test-zone3, test-zone1} 2020-11-29T16:03:43.595 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-18 zones={test-zone3, test-zone2} 2020-11-29T16:03:43.606 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:03:43.654 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:43.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:43.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:43.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:43.656 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:03:43.666 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:03:43.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:03:43.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:03:43.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:03:43.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:43.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:43.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:03:43.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:43.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:44.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:03:44.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:44.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:44.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:03:44.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:44.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:44.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:03:44.702 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-18 zones={test-zone3, test-zone2} 2020-11-29T16:03:45.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:45.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:45.352 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:45.401 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:45.401 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:45.632 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:45.628+0000 7f2a9b61e080 1 waiting for incremental sync to catch up: 2020-11-29T16:03:45.633 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.26.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:45.633 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.32.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:50.636 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:50.632+0000 7f2a9b61e080 1 waiting for incremental sync to catch up: 2020-11-29T16:03:50.637 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.26.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:50.637 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.32.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:55.635+0000 7f2a9b61e080 1 bucket sync caught up with source: 2020-11-29T16:03:55.641 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.32.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:55.641 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.32.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:55.641 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:55.635+0000 7f2a9b61e080 0 bucket checkpoint complete 2020-11-29T16:03:55.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:55.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:55.675 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:55.723 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:55.724 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:55.940+0000 7ffa5406b080 1 bucket sync caught up with source: 2020-11-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000011.32.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000011.32.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:55.941 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:55.940+0000 7ffa5406b080 0 bucket checkpoint complete 2020-11-29T16:03:55.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:55.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:55.970 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:56.019 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:56.019 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:56.255 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:56.251+0000 7f7809641080 1 bucket sync caught up with source: 2020-11-29T16:03:56.256 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.32.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:56.256 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.32.5, , 00000000003.19.9, , 00000000009.28.5, , , , , ] 2020-11-29T16:03:56.256 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:03:56.251+0000 7f7809641080 0 bucket checkpoint complete 2020-11-29T16:03:56.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:56.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:03:56.290 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:03:56.341 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:03:56.341 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:03:56.569 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:03:56.568+0000 7fc1f2232080 1 waiting for incremental sync to catch up: 2020-11-29T16:03:56.570 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.18.5, , 00000000010.35.5, , 00000000008.50.5, , , , , ] 2020-11-29T16:03:56.570 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.18.5, , 00000000011.48.5, , 00000000008.50.5, , , , , ] 2020-11-29T16:04:01.574 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:01.572+0000 7fc1f2232080 1 bucket sync caught up with source: 2020-11-29T16:04:01.574 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.18.5, , 00000000011.48.5, , 00000000008.50.5, , , , , ] 2020-11-29T16:04:01.575 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.18.5, , 00000000011.48.5, , 00000000008.50.5, , , , , ] 2020-11-29T16:04:01.575 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:01.572+0000 7fc1f2232080 0 bucket checkpoint complete 2020-11-29T16:04:01.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:01.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:01.602 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:01.648 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:01.648 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:01.877 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:01.876+0000 7f9f5b208080 1 bucket sync caught up with source: 2020-11-29T16:04:01.878 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.18.5, , 00000000011.48.5, , 00000000008.50.5, , , , , ] 2020-11-29T16:04:01.878 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.18.5, , 00000000011.48.5, , 00000000008.50.5, , , , , ] 2020-11-29T16:04:01.878 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:01.876+0000 7f9f5b208080 0 bucket checkpoint complete 2020-11-29T16:04:01.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:01.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:01.912 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:01.999 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:01.999 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:02.227 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:02.224+0000 7f2f3516a080 1 bucket sync caught up with source: 2020-11-29T16:04:02.227 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.20.5, , 00000000004.46.5, , 00000000015.44.5, , , , , ] 2020-11-29T16:04:02.228 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.20.5, , 00000000004.46.5, , 00000000015.44.5, , , , , ] 2020-11-29T16:04:02.228 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:02.224+0000 7f2f3516a080 0 bucket checkpoint complete 2020-11-29T16:04:02.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:02.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:02.254 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:02.302 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:02.303 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:02.562 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:02.559+0000 7f7c9e874080 1 bucket sync caught up with source: 2020-11-29T16:04:02.562 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000004.20.5, , 00000000004.46.5, , 00000000015.44.5, , , , , ] 2020-11-29T16:04:02.562 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000004.20.5, , 00000000004.46.5, , 00000000015.44.5, , , , , ] 2020-11-29T16:04:02.563 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:02.559+0000 7f7c9e874080 0 bucket checkpoint complete 2020-11-29T16:04:02.592 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-16 zones={test-zone2, test-zone1} 2020-11-29T16:04:02.604 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:02.610 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:02.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:02.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:02.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:02.612 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:02.620 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:02.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:02.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:02.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:02.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:02.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:02.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:02.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:02.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:02.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:03.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:03.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:03.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:03.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:03.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:03.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:03.650 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-16 zones={test-zone2, test-zone1} 2020-11-29T16:04:03.650 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-16 zones={test-zone3, test-zone1} 2020-11-29T16:04:03.660 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:03.706 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:03.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:03.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:03.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:03.708 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:03.715 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:03.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:03.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:03.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:03.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:03.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:03.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:03.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:03.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:04.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:04.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:04.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:04.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:04.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:04.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:04.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:04.745 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-16 zones={test-zone3, test-zone1} 2020-11-29T16:04:04.746 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-16 zones={test-zone3, test-zone2} 2020-11-29T16:04:04.757 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:04.802 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:04.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:04.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:04.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:04.805 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:04.814 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:04.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:04.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:04.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:04.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:04.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:04.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:05.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:05.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:05.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:05.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:05.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:05.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:05.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:05.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:05.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:05.841 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-16 zones={test-zone3, test-zone2} 2020-11-29T16:04:05.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:05.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:05.843 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:05.887 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:05.887 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:06.118 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:06.115+0000 7fd22cd61080 1 bucket sync caught up with source: 2020-11-29T16:04:06.118 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.31.5, , 00000000004.15.5, , 00000000008.29.5, , , , , ] 2020-11-29T16:04:06.119 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.31.5, , 00000000004.15.5, , 00000000008.29.5, , , , , ] 2020-11-29T16:04:06.119 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:06.115+0000 7fd22cd61080 0 bucket checkpoint complete 2020-11-29T16:04:06.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:06.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:06.152 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:06.198 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:06.198 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:06.426 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:06.424+0000 7f2c4de6d080 1 bucket sync caught up with source: 2020-11-29T16:04:06.426 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000011.31.5, , 00000000004.15.5, , 00000000008.29.5, , , , , ] 2020-11-29T16:04:06.426 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000011.31.5, , 00000000004.15.5, , 00000000008.29.5, , , , , ] 2020-11-29T16:04:06.427 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:06.424+0000 7f2c4de6d080 0 bucket checkpoint complete 2020-11-29T16:04:06.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:06.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:06.458 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:06.506 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:06.507 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:06.747 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:06.743+0000 7efdea861080 1 bucket sync caught up with source: 2020-11-29T16:04:06.747 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.31.5, , 00000000004.15.5, , 00000000008.29.5, , , , , ] 2020-11-29T16:04:06.747 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.31.5, , 00000000004.15.5, , 00000000008.29.5, , , , , ] 2020-11-29T16:04:06.748 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:06.743+0000 7efdea861080 0 bucket checkpoint complete 2020-11-29T16:04:06.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:06.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:06.874 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:06.922 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:06.922 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:07.161 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:07.160+0000 7fdcbb879080 1 bucket sync caught up with source: 2020-11-29T16:04:07.162 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.19.5, , 00000000011.35.5, , 00000000010.38.5, , , , , ] 2020-11-29T16:04:07.162 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.19.5, , 00000000011.35.5, , 00000000010.38.5, , , , , ] 2020-11-29T16:04:07.162 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:07.160+0000 7fdcbb879080 0 bucket checkpoint complete 2020-11-29T16:04:07.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:07.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:07.194 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:07.240 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:07.241 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:07.475 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:07.472+0000 7fbc579fb080 1 bucket sync caught up with source: 2020-11-29T16:04:07.475 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.19.5, , 00000000011.35.5, , 00000000010.38.5, , , , , ] 2020-11-29T16:04:07.476 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.19.5, , 00000000011.35.5, , 00000000010.38.5, , , , , ] 2020-11-29T16:04:07.476 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:07.472+0000 7fbc579fb080 0 bucket checkpoint complete 2020-11-29T16:04:07.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:07.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:07.510 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:07.598 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:07.598 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:07.842 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:07.840+0000 7fa0a3d4b080 1 bucket sync caught up with source: 2020-11-29T16:04:07.842 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.17.5, , 00000000006.23.5, , 00000000015.46.5, , , , , ] 2020-11-29T16:04:07.842 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.17.5, , 00000000006.23.5, , 00000000015.46.5, , , , , ] 2020-11-29T16:04:07.843 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:07.840+0000 7fa0a3d4b080 0 bucket checkpoint complete 2020-11-29T16:04:07.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:07.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:07.871 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:07.915 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:07.915 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:08.152 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:08.147+0000 7fdc3437b080 1 bucket sync caught up with source: 2020-11-29T16:04:08.152 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000004.17.5, , 00000000006.23.5, , 00000000015.46.5, , , , , ] 2020-11-29T16:04:08.153 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000004.17.5, , 00000000006.23.5, , 00000000015.46.5, , , , , ] 2020-11-29T16:04:08.153 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:08.147+0000 7fdc3437b080 0 bucket checkpoint complete 2020-11-29T16:04:08.184 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-17 zones={test-zone2, test-zone1} 2020-11-29T16:04:08.195 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:08.202 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:08.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:08.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:08.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:08.204 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:08.211 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:08.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:08.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:08.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:08.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:08.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:08.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:08.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:08.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:08.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:08.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:08.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:08.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:08.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:08.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:09.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:09.214 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-17 zones={test-zone2, test-zone1} 2020-11-29T16:04:09.215 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-17 zones={test-zone3, test-zone1} 2020-11-29T16:04:09.224 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:09.229 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:09.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:09.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:09.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:09.231 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:09.278 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:09.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:09.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:09.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:09.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:09.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:09.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:09.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:09.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:09.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:09.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:09.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:09.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:10.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:10.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:10.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:10.277 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-17 zones={test-zone3, test-zone1} 2020-11-29T16:04:10.278 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-17 zones={test-zone3, test-zone2} 2020-11-29T16:04:10.289 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:10.293 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:10.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:10.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:10.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:10.295 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:10.300 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:10.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:10.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:10.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:10.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:10.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:10.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:10.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:10.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:10.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:10.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:10.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:10.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:11.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:11.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:11.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:11.285 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-17 zones={test-zone3, test-zone2} 2020-11-29T16:04:11.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:11.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:11.287 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:11.333 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:11.333 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:11.579+0000 7efeed193080 1 bucket sync caught up with source: 2020-11-29T16:04:11.583 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.32.5, , 00000000004.33.5, , 00000000008.31.5, , , , , ] 2020-11-29T16:04:11.583 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.32.5, , 00000000004.33.5, , 00000000008.31.5, , , , , ] 2020-11-29T16:04:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:11.579+0000 7efeed193080 0 bucket checkpoint complete 2020-11-29T16:04:11.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:11.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:11.618 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:11.667 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:11.668 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:11.912 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:11.908+0000 7fa106a37080 1 bucket sync caught up with source: 2020-11-29T16:04:11.912 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000011.32.5, , 00000000004.33.5, , 00000000008.31.5, , , , , ] 2020-11-29T16:04:11.912 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000011.32.5, , 00000000004.33.5, , 00000000008.31.5, , , , , ] 2020-11-29T16:04:11.912 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:11.908+0000 7fa106a37080 0 bucket checkpoint complete 2020-11-29T16:04:11.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:11.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:11.943 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:11.990 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:11.990 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:12.239 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:12.235+0000 7f316bb12080 1 bucket sync caught up with source: 2020-11-29T16:04:12.239 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.32.5, , 00000000004.33.5, , 00000000008.31.5, , , , , ] 2020-11-29T16:04:12.240 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.32.5, , 00000000004.33.5, , 00000000008.31.5, , , , , ] 2020-11-29T16:04:12.240 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:12.235+0000 7f316bb12080 0 bucket checkpoint complete 2020-11-29T16:04:12.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:12.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:12.362 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:12.407 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:12.407 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:12.638 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:12.636+0000 7f97c34d6080 1 bucket sync caught up with source: 2020-11-29T16:04:12.639 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.17.5, , 00000000011.54.5, , 00000000008.56.5, , , , , ] 2020-11-29T16:04:12.639 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.17.5, , 00000000011.54.5, , 00000000008.56.5, , , , , ] 2020-11-29T16:04:12.639 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:12.636+0000 7f97c34d6080 0 bucket checkpoint complete 2020-11-29T16:04:12.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:12.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:12.673 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:12.760 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:12.760 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:13.002 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:13.000+0000 7f4870e7a080 1 bucket sync caught up with source: 2020-11-29T16:04:13.003 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.17.5, , 00000000011.54.5, , 00000000008.56.5, , , , , ] 2020-11-29T16:04:13.003 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.17.5, , 00000000011.54.5, , 00000000008.56.5, , , , , ] 2020-11-29T16:04:13.003 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:13.000+0000 7f4870e7a080 0 bucket checkpoint complete 2020-11-29T16:04:13.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:13.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:13.035 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:13.080 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:13.081 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:13.316 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:13.312+0000 7f7269d2d080 1 bucket sync caught up with source: 2020-11-29T16:04:13.316 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000004.18.5, , 00000000004.58.5, , 00000000016.56.5, , , , , ] 2020-11-29T16:04:13.316 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000004.18.5, , 00000000004.58.5, , 00000000016.56.5, , , , , ] 2020-11-29T16:04:13.317 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:13.312+0000 7f7269d2d080 0 bucket checkpoint complete 2020-11-29T16:04:13.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:13.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:13.346 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:13.395 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:13.395 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:13.629 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:13.623+0000 7f58708fe080 1 bucket sync caught up with source: 2020-11-29T16:04:13.629 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000004.18.5, , 00000000004.58.5, , 00000000016.56.5, , , , , ] 2020-11-29T16:04:13.629 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000004.18.5, , 00000000004.58.5, , 00000000016.56.5, , , , , ] 2020-11-29T16:04:13.629 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:13.623+0000 7f58708fe080 0 bucket checkpoint complete 2020-11-29T16:04:13.660 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-18 zones={test-zone2, test-zone1} 2020-11-29T16:04:13.671 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:13.722 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:13.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:13.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:13.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:13.724 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:13.733 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:13.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:13.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:13.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:13.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:13.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:13.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:13.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:13.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:14.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:14.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:14.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:14.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:14.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:14.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:14.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:14.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-18 zones={test-zone2, test-zone1} 2020-11-29T16:04:14.787 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-18 zones={test-zone3, test-zone1} 2020-11-29T16:04:14.796 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:14.802 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:14.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:14.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:14.804 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:14.804 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:14.811 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:14.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:14.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:14.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:14.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:14.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:14.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:14.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:15.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:15.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:15.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:15.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:15.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:15.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:15.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:15.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:15.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-18 zones={test-zone3, test-zone1} 2020-11-29T16:04:15.807 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-18 zones={test-zone3, test-zone2} 2020-11-29T16:04:15.819 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:15.870 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:15.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:15.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:15.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:15.872 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:15.878 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:15.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-11-29T16:04:15.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-11-29T16:04:15.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-11-29T16:04:15.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:15.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:16.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:16.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:16.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:16.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-11-29T16:04:16.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:16.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:16.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-11-29T16:04:16.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:16.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:16.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-11-29T16:04:16.949 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-18 zones={test-zone3, test-zone2} 2020-11-29T16:04:16.950 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-11-29T16:04:16.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:04:16.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:04:16.995 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:04:17.027 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:17.028 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:17.238 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:04:17.272 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:04:17.272 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:04:17.272 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:04:17.272 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:04:17.273 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:04:17.273 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:04:17.273 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665856.991097_91.1", 2020-11-29T16:04:17.273 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:16.991097Z" 2020-11-29T16:04:17.274 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:04:17.274 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:04:17.274 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:04:17.274 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:04:17.274 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:04:17.275 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:04:17.275 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:04:17.275 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:04:17.275 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:04:17.276 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:04:17.277 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:04:17.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:04:17.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:04:17.278 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:04:17.315 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:17.316 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:17.579 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:04:17.580 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:04:17.580 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:04:17.580 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:04:17.580 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:04:17.581 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:04:17.581 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:17.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:17.581 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:04:17.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:04:17.582 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:04:17.582 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:04:17.582 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:04:17.582 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:04:17.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:04:17.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:04:17.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:04:17.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:04:17.584 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:17.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:17.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:17.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:04:17.585 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:04:17.585 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:04:17.585 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:04:17.585 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665856.991097_91.1", 2020-11-29T16:04:17.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:04:17.586 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:04:17.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:04:17.586 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.991097Z", 2020-11-29T16:04:17.586 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:17.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:17.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:17.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:04:17.587 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:04:17.588 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:04:17.588 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:04:17.588 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:04:17.588 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:04:17.588 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:04:17.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:04:17.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:04:17.589 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:17.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:17.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:17.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:04:17.590 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:04:17.590 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:04:17.590 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:04:17.590 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:04:17.591 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:04:17.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:04:17.591 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:04:17.591 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:04:17.591 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:17.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:17.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:17.592 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:04:17.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:17.592 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:04:17.593 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:04:17.593 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:04:17.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:17.593 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:04:17.594 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:04:17.595 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:04:17.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:04:17.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:04:17.596 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:04:17.638 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:17.638 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:17.881 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:04:17.881 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:04:17.882 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:04:17.882 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:04:17.882 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:04:17.883 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:04:17.883 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:04:17.884 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:04:17.884 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:04:17.884 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:04:17.885 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:04:17.885 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:04:17.886 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:04:17.886 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:04:17.887 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:04:17.887 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:04:17.887 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:04:17.888 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:04:17.888 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:04:17.889 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:04:17.889 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:04:17.890 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:04:17.890 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:04:17.890 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:04:17.891 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:04:17.891 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665856.991097_91.1", 2020-11-29T16:04:17.892 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:04:17.892 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:04:17.892 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:04:17.893 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.991097Z", 2020-11-29T16:04:17.893 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:04:17.893 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:04:17.894 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:04:17.894 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:04:17.895 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:04:17.895 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:04:17.896 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:04:17.896 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:04:17.897 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:04:17.897 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:04:17.897 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:04:17.898 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:04:17.898 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:04:17.899 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:04:17.899 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:04:17.899 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:04:17.900 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:04:17.900 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:04:17.900 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:04:17.901 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:04:17.901 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:04:17.902 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:04:17.902 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:04:17.902 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:04:17.902 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:04:17.903 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:04:17.903 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:04:17.903 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:04:17.903 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:04:17.904 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:04:17.904 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:04:17.904 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:04:17.904 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:04:17.904 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:04:17.906 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:04:17.906 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:04:17.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:04:17.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:04:17.908 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:04:17.948 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:17.948 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:18.221 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:04:18.222 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:04:18.222 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:04:18.222 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:04:18.223 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:04:18.223 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:04:18.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:18.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:18.224 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:04:18.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:04:18.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:04:18.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:04:18.225 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:04:18.225 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:04:18.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:04:18.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:04:18.225 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:04:18.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:04:18.226 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:18.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:18.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:18.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:04:18.227 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:04:18.227 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:04:18.227 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:04:18.228 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665856.991097_91.1", 2020-11-29T16:04:18.228 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:04:18.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:04:18.228 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:04:18.229 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.991097Z", 2020-11-29T16:04:18.229 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:18.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:18.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:18.229 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:04:18.230 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:04:18.230 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:04:18.230 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:04:18.230 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:04:18.231 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:04:18.231 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:04:18.231 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:04:18.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:04:18.232 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:18.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:18.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:18.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:04:18.232 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:04:18.233 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:04:18.233 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:04:18.233 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:04:18.233 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:04:18.234 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:04:18.234 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:04:18.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:04:18.234 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:04:18.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:18.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:18.235 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:04:18.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:04:18.235 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:04:18.236 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:04:18.236 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:04:18.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:04:18.236 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:04:18.237 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:04:18.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:18.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:18.288 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:18.335 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:18.335 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:18.560 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:18.555+0000 7f6fe6583080 1 waiting for incremental sync to catch up: 2020-11-29T16:04:18.560 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:04:18.561 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.37.10, , , , , ] 2020-11-29T16:04:23.571 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:23.559+0000 7f6fe6583080 1 bucket sync caught up with source: 2020-11-29T16:04:23.571 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.37.10, , , , , ] 2020-11-29T16:04:23.571 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.37.10, , , , , ] 2020-11-29T16:04:23.572 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:23.559+0000 7f6fe6583080 0 bucket checkpoint complete 2020-11-29T16:04:23.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:23.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:23.598 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:23.648 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:23.648 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:23.878 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:23.876+0000 7f8f7237d080 1 waiting for incremental sync to catch up: 2020-11-29T16:04:23.879 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000002.37.10, , , , , ] 2020-11-29T16:04:23.879 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2020-11-29T16:04:28.883 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:28.879+0000 7f8f7237d080 1 waiting for incremental sync to catch up: 2020-11-29T16:04:28.883 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000002.37.10, , , , , ] 2020-11-29T16:04:28.884 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2020-11-29T16:04:33.887 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:33.883+0000 7f8f7237d080 1 waiting for incremental sync to catch up: 2020-11-29T16:04:33.887 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000002.37.10, , , , , ] 2020-11-29T16:04:33.887 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2020-11-29T16:04:38.891 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:38.887+0000 7f8f7237d080 1 bucket sync caught up with source: 2020-11-29T16:04:38.891 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.41.9, , , , , ] 2020-11-29T16:04:38.891 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.41.9, , , , , ] 2020-11-29T16:04:38.892 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:38.887+0000 7f8f7237d080 0 bucket checkpoint complete 2020-11-29T16:04:38.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:38.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:38.924 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:38.971 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:38.971 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:39.226 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:39.223+0000 7f385db26080 1 bucket sync caught up with source: 2020-11-29T16:04:39.227 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.43.9, , , , , ] 2020-11-29T16:04:39.227 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , , , ] 2020-11-29T16:04:39.227 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:39.223+0000 7f385db26080 0 bucket checkpoint complete 2020-11-29T16:04:39.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:39.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:39.344 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:39.392 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:39.393 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:39.625 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:39.619+0000 7f5072147080 1 bucket sync caught up with source: 2020-11-29T16:04:39.625 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , , , ] 2020-11-29T16:04:39.625 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , , , ] 2020-11-29T16:04:39.626 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:39.619+0000 7f5072147080 0 bucket checkpoint complete 2020-11-29T16:04:39.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:39.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:39.658 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:39.706 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:39.707 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:39.951 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:39.947+0000 7fc36f27d080 1 bucket sync caught up with source: 2020-11-29T16:04:39.951 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , , , ] 2020-11-29T16:04:39.952 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , , , ] 2020-11-29T16:04:39.952 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:39.947+0000 7fc36f27d080 0 bucket checkpoint complete 2020-11-29T16:04:39.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:39.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:39.984 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:40.031 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:40.031 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:40.268 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:40.263+0000 7fe7bff87080 1 bucket sync caught up with source: 2020-11-29T16:04:40.268 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.67.9, , , , , ] 2020-11-29T16:04:40.268 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.67.9, , , , , ] 2020-11-29T16:04:40.269 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:04:40.263+0000 7fe7bff87080 0 bucket checkpoint complete 2020-11-29T16:04:40.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:40.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:40.300 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:40.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:40.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:40.596 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:40.591+0000 7fd28ef01080 1 bucket sync caught up with source: 2020-11-29T16:04:40.597 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.67.9, , , , , ] 2020-11-29T16:04:40.597 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.67.9, , , , , ] 2020-11-29T16:04:40.597 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:40.591+0000 7fd28ef01080 0 bucket checkpoint complete 2020-11-29T16:04:40.626 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-19 zones={test-zone2, test-zone1} 2020-11-29T16:04:40.634 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:40.686 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:40.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:40.693 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:40.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:40.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:40.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:40.897 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-19 zones={test-zone2, test-zone1} 2020-11-29T16:04:40.898 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-19 zones={test-zone3, test-zone1} 2020-11-29T16:04:40.907 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:40.958 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:40.958 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:40.963 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:41.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:41.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:41.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:41.152 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-19 zones={test-zone3, test-zone1} 2020-11-29T16:04:41.153 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-19 zones={test-zone3, test-zone2} 2020-11-29T16:04:41.163 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:04:41.214 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:41.215 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:04:41.222 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:04:41.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:41.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:41.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:04:41.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-19 zones={test-zone3, test-zone2} 2020-11-29T16:04:42.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:42.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:04:42.900 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:04:42.946 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:04:42.946 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:04:43.186 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:43.183+0000 7fa81b3a7080 1 waiting for incremental sync to catch up: 2020-11-29T16:04:43.186 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000009.24.5, , ] 2020-11-29T16:04:43.186 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000027.67.11, , ] 2020-11-29T16:04:48.190 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:48.187+0000 7fa81b3a7080 1 waiting for incremental sync to catch up: 2020-11-29T16:04:48.190 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000009.24.5, , ] 2020-11-29T16:04:48.191 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000027.67.11, , ] 2020-11-29T16:04:53.193 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:53.191+0000 7fa81b3a7080 1 waiting for incremental sync to catch up: 2020-11-29T16:04:53.194 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000009.24.5, , ] 2020-11-29T16:04:53.194 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000027.67.11, , ] 2020-11-29T16:04:58.197 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:04:58.194+0000 7fa81b3a7080 1 waiting for incremental sync to catch up: 2020-11-29T16:04:58.197 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.41.9, , , 00000000009.24.5, , ] 2020-11-29T16:04:58.198 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000027.67.11, , ] 2020-11-29T16:05:03.201 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:05:03.198+0000 7fa81b3a7080 1 bucket sync caught up with source: 2020-11-29T16:05:03.201 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.43.9, , , 00000000038.101.5, , ] 2020-11-29T16:05:03.201 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000027.67.11, , ] 2020-11-29T16:05:03.202 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:05:03.198+0000 7fa81b3a7080 0 bucket checkpoint complete 2020-11-29T16:05:03.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:05:03.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:05:03.235 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:05:03.278 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:05:03.278 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:05:03.510 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:03.507+0000 7ff3ae866080 1 bucket sync caught up with source: 2020-11-29T16:05:03.510 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000005.43.9, , , 00000000044.112.9, , ] 2020-11-29T16:05:03.510 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000044.112.9, , ] 2020-11-29T16:05:03.510 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:03.507+0000 7ff3ae866080 0 bucket checkpoint complete 2020-11-29T16:05:03.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:05:03.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:05:03.537 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:05:03.585 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:05:03.585 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:05:03.828 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:05:03.826+0000 7f94ff0d9080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:03.829 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.43.9, , , 00000000030.76.5, , ] 2020-11-29T16:05:03.829 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000044.112.9, , ] 2020-11-29T16:05:08.832 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:05:08.826+0000 7f94ff0d9080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:08.833 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.43.9, , , 00000000030.76.5, , ] 2020-11-29T16:05:08.833 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000044.112.9, , ] 2020-11-29T16:05:13.834 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:05:13.830+0000 7f94ff0d9080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:13.835 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.43.9, , , 00000000030.76.5, , ] 2020-11-29T16:05:13.835 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000044.112.9, , ] 2020-11-29T16:05:18.839 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:05:18.834+0000 7f94ff0d9080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:18.839 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.43.9, , , 00000000030.76.5, , ] 2020-11-29T16:05:18.839 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000044.112.9, , ] 2020-11-29T16:05:23.843 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:05:23.838+0000 7f94ff0d9080 1 bucket sync caught up with source: 2020-11-29T16:05:23.843 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.43.9, , , 00000000044.112.9, , ] 2020-11-29T16:05:23.844 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.43.9, , , 00000000044.112.9, , ] 2020-11-29T16:05:23.844 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:05:23.838+0000 7f94ff0d9080 0 bucket checkpoint complete 2020-11-29T16:05:23.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:05:23.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:05:23.880 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:05:23.928 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:05:23.929 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:05:24.156 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:24.150+0000 7f4001831080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:24.157 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.118.8, , ] 2020-11-29T16:05:24.157 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:05:29.161 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:29.154+0000 7f4001831080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:29.161 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.118.8, , ] 2020-11-29T16:05:29.161 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:05:34.165 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:34.162+0000 7f4001831080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:34.165 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.118.8, , ] 2020-11-29T16:05:34.166 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:05:39.169 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:39.166+0000 7f4001831080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:39.169 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.118.8, , ] 2020-11-29T16:05:39.170 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:05:44.173 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:44.166+0000 7f4001831080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:44.173 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.118.8, , ] 2020-11-29T16:05:44.173 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:05:49.177 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:49.170+0000 7f4001831080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:49.177 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.118.8, , ] 2020-11-29T16:05:49.177 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:05:54.181 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:54.174+0000 7f4001831080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:54.181 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.118.8, , ] 2020-11-29T16:05:54.181 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:05:59.185 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:05:59.178+0000 7f4001831080 1 waiting for incremental sync to catch up: 2020-11-29T16:05:59.185 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000045.118.8, , ] 2020-11-29T16:05:59.185 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:06:04.188 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:04.182+0000 7f4001831080 1 bucket sync caught up with source: 2020-11-29T16:06:04.189 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:06:04.189 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:06:04.189 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:04.182+0000 7f4001831080 0 bucket checkpoint complete 2020-11-29T16:06:04.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:04.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:04.223 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:04.268 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:04.268 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:04.503 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:04.498+0000 7f066b21d080 1 bucket sync caught up with source: 2020-11-29T16:06:04.504 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:06:04.504 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.65.9, , , 00000000047.121.9, , ] 2020-11-29T16:06:04.504 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:04.498+0000 7f066b21d080 0 bucket checkpoint complete 2020-11-29T16:06:04.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:04.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:04.537 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:04.624 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:04.625 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:04.876 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:04.870+0000 7f7255ccf080 1 bucket sync caught up with source: 2020-11-29T16:06:04.876 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000004.67.9, , , 00000000049.122.9, , ] 2020-11-29T16:06:04.877 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000004.67.9, , , 00000000049.122.9, , ] 2020-11-29T16:06:04.877 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:04.870+0000 7f7255ccf080 0 bucket checkpoint complete 2020-11-29T16:06:04.903 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:04.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:04.905 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:04.950 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:04.951 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:05.184 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:05.182+0000 7f2a5641e080 1 bucket sync caught up with source: 2020-11-29T16:06:05.185 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.67.9, , , 00000000049.122.9, , ] 2020-11-29T16:06:05.185 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.67.9, , , 00000000049.122.9, , ] 2020-11-29T16:06:05.185 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:05.182+0000 7f2a5641e080 0 bucket checkpoint complete 2020-11-29T16:06:05.220 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-19 zones={test-zone2, test-zone1} 2020-11-29T16:06:05.233 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:06:05.243 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:06:05.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.256 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:06:05.311 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:06:05.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:05.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:05.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:05.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:05.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:05.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:05.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:05.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:05.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:05.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:06.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:07.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:08.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:09.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:10.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:11.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:12.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:13.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.328 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-19 zones={test-zone2, test-zone1} 2020-11-29T16:06:14.328 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-19 zones={test-zone3, test-zone1} 2020-11-29T16:06:14.336 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:06:14.342 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:06:14.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:06:14.363 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:06:14.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:14.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:14.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:14.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:14.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:14.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:15.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:16.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:17.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:18.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:19.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:20.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:21.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:22.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:23.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:23.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:23.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:23.323 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-19 zones={test-zone3, test-zone1} 2020-11-29T16:06:23.323 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-19 zones={test-zone3, test-zone2} 2020-11-29T16:06:23.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:06:23.342 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:06:23.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.354 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:06:23.365 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-11-29T16:06:23.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:23.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:23.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:23.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-11-29T16:06:23.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:23.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:23.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:23.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:23.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:24.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:25.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:26.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:27.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:28.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:29.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:30.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:31.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:32.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:32.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:32.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:32.315 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-19 zones={test-zone3, test-zone2} 2020-11-29T16:06:32.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:32.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:32.318 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:32.358 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:32.359 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:32.896 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:06:32.896 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:32.896 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-29T16:06:32.896 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:05:21.320012Z" 2020-11-29T16:06:32.897 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:32.897 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:32.897 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:06:32.897 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:06:32.897 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:32.898 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:32.898 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:32.898 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:32.898 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:32.899 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:32.899 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:32.899 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:06:32.899 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:32.899 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:06:32.900 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:06:32.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:32.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:32.902 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:32.945 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:32.946 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:33.198 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:06:33.199 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:06:33.199 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:06:33.199 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:06:33.199 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:06:33.200 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 6162550666273865027 2020-11-29T16:06:33.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:33.200 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:06:33.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:33.201 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:06:33.201 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:33.201 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:33.201 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-29T16:06:33.202 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:33.202 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:33.202 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:33.202 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:05:21.320012Z" 2020-11-29T16:06:33.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:33.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:33.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:33.203 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:06:33.204 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:33.204 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:33.204 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:06:33.204 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:33.205 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:33.205 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:33.205 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:06:33.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:06:33.206 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:33.207 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2020-11-29T16:06:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:33.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:06:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:33.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:33.210 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:06:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:06:33.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:33.212 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:06:33.212 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:06:33.212 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:06:33.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:33.213 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:06:33.214 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:06:33.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:33.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:33.216 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:33.254 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:33.255 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:33.508 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:06:33.509 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:33.509 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-29T16:06:33.509 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:05:21.320012Z" 2020-11-29T16:06:33.509 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:33.509 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:33.510 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:06:33.510 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:06:33.510 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:33.510 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:33.511 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:33.511 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:33.511 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:33.511 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:33.511 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:33.512 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:06:33.512 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:33.512 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:06:33.513 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:06:33.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:33.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:33.515 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:33.601 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:33.601 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:33.862 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:06:33.862 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:06:33.862 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:06:33.863 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:06:33.863 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:06:33.863 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 5268709521422156746 2020-11-29T16:06:33.863 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:33.864 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:06:33.864 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:33.864 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:06:33.864 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:33.864 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:33.865 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-29T16:06:33.865 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:33.865 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:33.865 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:33.865 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:05:21.320012Z" 2020-11-29T16:06:33.866 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:33.866 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:33.866 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:33.866 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:06:33.867 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:33.867 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:33.867 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:06:33.867 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:33.867 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:33.868 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:33.868 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:06:33.868 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:33.868 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:33.868 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:33.869 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:06:33.869 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:33.869 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:33.869 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:33.870 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:33.870 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:33.870 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:33.870 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000" 2020-11-29T16:06:33.870 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:33.871 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:33.871 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:33.871 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:06:33.871 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:33.871 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:33.872 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:33.872 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:33.872 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:33.872 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:33.872 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:06:33.873 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:33.873 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:33.873 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:06:33.873 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:33.873 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:06:33.874 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:06:33.874 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:06:33.874 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:33.874 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:06:33.875 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:06:33.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:33.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:33.877 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:33.957 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:33.958 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:34.210 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:06:34.211 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:34.211 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-29T16:06:34.211 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:05:21.320012Z" 2020-11-29T16:06:34.211 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:34.211 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:34.212 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:06:34.212 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:06:34.212 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:34.212 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:34.213 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:34.213 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:34.213 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:34.213 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:34.214 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:34.214 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:06:34.214 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:34.214 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:06:34.215 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:06:34.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:34.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:34.217 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:34.259 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:34.259 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:34.535 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:06:34.536 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:06:34.536 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:06:34.536 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:06:34.536 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:06:34.536 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 16089441476439148043 2020-11-29T16:06:34.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:34.537 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:06:34.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:34.537 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:06:34.538 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:34.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:34.538 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001177", 2020-11-29T16:06:34.538 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:34.538 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:34.539 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:34.539 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:05:21.320012Z" 2020-11-29T16:06:34.539 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:34.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:34.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:34.540 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:06:34.540 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:34.540 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:34.540 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:06:34.541 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:34.541 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:34.541 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:34.541 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:06:34.541 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:34.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:34.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:34.542 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:06:34.542 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:34.542 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:34.543 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:34.543 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:34.543 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:34.543 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:34.544 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2020-11-29T16:06:34.544 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:34.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:34.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:34.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:06:34.545 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:34.545 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:34.545 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:34.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:34.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:34.546 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:34.546 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:06:34.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:34.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:34.547 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:06:34.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:34.547 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:06:34.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:06:34.547 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:06:34.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:34.548 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:06:34.549 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:06:34.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:34.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:34.551 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:34.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:34.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:34.888 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:06:34.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:34.889 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-29T16:06:34.889 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:05:43.464887Z" 2020-11-29T16:06:34.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:34.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:34.890 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:06:34.890 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:06:34.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:34.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:34.891 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:34.891 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-11-29T16:06:34.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:34.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:34.892 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:06:34.892 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:06:34.892 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:34.893 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:06:34.894 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:06:34.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:34.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:34.895 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:34.937 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:34.938 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:35.211 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:06:35.211 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:06:35.212 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:06:35.212 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:06:35.213 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:06:35.213 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 201246450793848048 2020-11-29T16:06:35.213 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.214 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:06:35.214 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:35.215 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:06:35.215 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:35.215 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:35.216 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-29T16:06:35.216 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:35.217 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:35.217 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:35.217 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:05:43.464887Z" 2020-11-29T16:06:35.218 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.218 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.219 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:35.219 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:06:35.220 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:35.220 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:35.220 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:06:35.221 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:35.221 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:35.222 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:35.222 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:06:35.222 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.223 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.223 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:35.224 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:06:35.224 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:35.224 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:35.225 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:35.225 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:35.226 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:35.226 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:35.227 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000" 2020-11-29T16:06:35.227 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.227 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.228 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:35.228 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:06:35.229 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:35.229 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:35.230 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:06:35.230 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:35.230 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:35.231 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:35.231 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:06:35.232 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.232 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.232 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:06:35.233 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.233 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:06:35.234 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:06:35.234 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:06:35.235 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.235 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:06:35.237 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:06:35.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:35.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:35.239 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:35.278 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:35.278 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:35.538 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:06:35.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:35.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-29T16:06:35.539 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:05:43.464887Z" 2020-11-29T16:06:35.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:35.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:35.540 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:06:35.540 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:06:35.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:35.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:35.541 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:35.541 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-11-29T16:06:35.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:06:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:06:35.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:35.542 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:06:35.543 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:06:35.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:35.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:35.545 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:35.587 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:35.588 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:35.846 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:06:35.846 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:06:35.847 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:06:35.847 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:06:35.847 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:06:35.847 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 16876182154419717581 2020-11-29T16:06:35.847 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.848 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:06:35.848 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:35.848 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:06:35.848 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:35.848 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:35.849 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-29T16:06:35.849 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:35.849 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:35.849 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:35.849 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:05:43.464887Z" 2020-11-29T16:06:35.850 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.850 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.850 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:35.850 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:06:35.851 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:35.851 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:35.851 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:06:35.851 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:35.851 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:35.852 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:35.852 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:06:35.852 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.852 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.852 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:35.853 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:06:35.853 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:35.853 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:35.853 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:35.853 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:35.854 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:35.854 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:35.854 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000" 2020-11-29T16:06:35.854 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.854 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.855 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:35.855 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:06:35.855 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:35.855 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:35.855 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:06:35.856 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:35.856 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:35.856 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:35.856 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:06:35.856 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.857 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.857 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:06:35.857 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:35.857 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:06:35.857 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:06:35.858 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:06:35.858 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:35.858 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:06:35.859 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:06:35.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:35.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:35.861 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:35.942 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:35.942 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:36.180 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:06:36.180 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.181 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-29T16:06:36.181 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:05:43.471301Z" 2020-11-29T16:06:36.181 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.181 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.181 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:06:36.182 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.461995Z" 2020-11-29T16:06:36.182 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.182 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.182 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:36.183 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:36.183 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.183 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.183 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:36.183 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:57.000149Z" 2020-11-29T16:06:36.184 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:36.184 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:06:36.185 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:06:36.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:36.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:36.186 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:36.270 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:36.270 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:36.528 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:06:36.528 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:06:36.529 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:06:36.529 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:06:36.529 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:06:36.529 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 3114293326551352523 2020-11-29T16:06:36.529 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.530 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:06:36.530 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.530 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:06:36.530 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:36.531 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:36.531 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-29T16:06:36.531 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:36.531 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:36.531 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:36.532 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:05:43.471301Z" 2020-11-29T16:06:36.532 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:36.532 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.532 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.532 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:06:36.533 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:36.533 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:36.533 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:06:36.533 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:36.533 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:36.534 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:36.534 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:59.461995Z" 2020-11-29T16:06:36.534 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:36.534 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.534 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.535 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:06:36.535 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:36.535 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:36.535 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:36.535 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:36.536 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:36.536 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:36.536 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000" 2020-11-29T16:06:36.536 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:36.536 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.537 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.537 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:06:36.537 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:36.537 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:06:36.537 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:36.538 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:36.538 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:36.538 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:36.538 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:03:57.000149Z" 2020-11-29T16:06:36.539 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:36.539 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:36.539 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:06:36.539 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.539 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:06:36.540 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:06:36.540 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:06:36.540 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:36.540 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:06:36.541 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:06:36.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:36.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:36.543 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:36.625 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:36.671 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:36.876 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:06:36.877 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.877 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-29T16:06:36.877 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:05:43.471301Z" 2020-11-29T16:06:36.877 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.878 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.878 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:06:36.878 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.461995Z" 2020-11-29T16:06:36.878 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.879 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.879 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:36.879 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:36.879 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:36.879 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:36.880 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:36.880 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:57.000149Z" 2020-11-29T16:06:36.880 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:36.880 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:06:36.881 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:06:36.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:36.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:36.883 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:36.921 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:36.921 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:37.181 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:06:37.181 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:06:37.182 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:06:37.182 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:06:37.182 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:06:37.182 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 2748347558345283993 2020-11-29T16:06:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:06:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:06:37.183 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-11-29T16:06:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:37.184 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:37.185 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:37.185 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:05:43.471301Z" 2020-11-29T16:06:37.185 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:37.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:06:37.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:37.186 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:37.186 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:06:37.186 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:37.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:37.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:37.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:59.461995Z" 2020-11-29T16:06:37.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.188 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:37.188 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:06:37.188 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:37.188 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:37.189 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-11-29T16:06:37.189 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:37.189 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:37.189 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:37.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2020-11-29T16:06:37.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.190 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:37.190 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:06:37.190 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:37.191 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:06:37.191 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:06:37.191 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:37.191 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:37.192 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:37.192 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:03:57.000149Z" 2020-11-29T16:06:37.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.192 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:06:37.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.193 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:06:37.193 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:06:37.193 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:06:37.193 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.194 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:06:37.195 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:06:37.195 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-11-29T16:06:37.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:37.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:37.239 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:37.277 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:37.278 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:37.470 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:06:37.501 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:06:37.502 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:37.502 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:06:37.503 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:06:37.503 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:37.503 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:37.504 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665997.230793_97.1", 2020-11-29T16:06:37.504 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:06:37.230793Z" 2020-11-29T16:06:37.505 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:37.505 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:37.506 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:06:37.506 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:37.507 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:37.507 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:37.508 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:06:37.508 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:37.509 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:37.509 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:06:37.511 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:06:37.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:37.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:37.513 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:37.554 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:37.554 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:37.795 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:06:37.796 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:06:37.796 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:06:37.796 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:06:37.796 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:06:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:06:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:06:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:37.798 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:06:37.798 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:37.798 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:37.798 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:06:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:06:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:06:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:06:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665997.230793_97.1", 2020-11-29T16:06:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.230793Z", 2020-11-29T16:06:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:06:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:06:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:06:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:37.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:37.805 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:06:37.805 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:37.806 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:37.806 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:06:37.806 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:37.806 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:37.806 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:37.807 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:06:37.807 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:37.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.807 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:06:37.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:37.808 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:06:37.808 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:06:37.808 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:06:37.809 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:37.809 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:06:37.811 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:06:37.812 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:06:37.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:37.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:37.813 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:37.855 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:37.856 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:38.123 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:06:38.123 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:06:38.123 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:06:38.124 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:06:38.124 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:06:38.124 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:06:38.124 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:38.125 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:38.125 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:06:38.125 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:38.125 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:06:38.125 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:38.126 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:06:38.126 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:06:38.126 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:38.126 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:06:38.126 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:38.127 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:06:38.127 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:38.127 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:38.127 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:38.127 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:38.128 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:06:38.128 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:38.128 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:06:38.128 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665997.230793_97.1", 2020-11-29T16:06:38.128 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:38.129 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:38.129 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:38.129 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.230793Z", 2020-11-29T16:06:38.129 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:38.130 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:38.130 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:38.130 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:38.130 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:06:38.130 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:38.131 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:06:38.131 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:06:38.131 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:38.131 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:38.131 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:38.132 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:06:38.132 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:38.132 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:38.132 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:38.132 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:38.133 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:06:38.133 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:38.133 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:06:38.133 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:06:38.134 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:38.134 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:38.134 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:38.134 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:06:38.134 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:38.135 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:38.135 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:38.135 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:06:38.135 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:38.136 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:06:38.136 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:06:38.136 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:06:38.136 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:38.136 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:06:38.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:06:38.138 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:06:38.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:38.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:38.139 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:38.179 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:38.180 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:38.548 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:06:38.548 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:06:38.549 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:06:38.549 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:06:38.549 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:06:38.549 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:06:38.550 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:38.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:38.550 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:06:38.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:38.551 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:06:38.551 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:38.551 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:38.551 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:06:38.551 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:38.552 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:06:38.552 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:38.552 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:06:38.552 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:38.552 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:38.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:38.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:38.553 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:06:38.553 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:38.553 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:38.554 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665997.230793_97.1", 2020-11-29T16:06:38.554 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:38.554 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:38.554 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:38.554 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.230793Z", 2020-11-29T16:06:38.555 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:38.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:38.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:38.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:38.556 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:06:38.556 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:38.556 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:38.556 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:06:38.556 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:38.557 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:38.557 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:38.557 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:06:38.557 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:38.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:38.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:38.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:38.558 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:06:38.558 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:38.559 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:38.559 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:06:38.559 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:38.559 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:38.559 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:38.560 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:06:38.560 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:38.560 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:38.560 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:38.560 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:06:38.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:38.561 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:06:38.561 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:06:38.561 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:06:38.561 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:38.562 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:06:38.562 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:06:38.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:38.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:38.619 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:38.668 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:38.668 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:38.920 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:38.917+0000 7fb6b72b4080 1 bucket sync caught up with source: 2020-11-29T16:06:38.921 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:38.921 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:38.921 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:38.917+0000 7fb6b72b4080 0 bucket checkpoint complete 2020-11-29T16:06:38.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:38.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:38.948 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:38.996 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:38.997 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:39.216 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:39.213+0000 7face48e8080 1 bucket sync caught up with source: 2020-11-29T16:06:39.216 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:39.216 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:39.217 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:39.213+0000 7face48e8080 0 bucket checkpoint complete 2020-11-29T16:06:39.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:39.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:39.244 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:39.293 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:39.294 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:39.550 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:39.545+0000 7f58c9984080 1 bucket sync caught up with source: 2020-11-29T16:06:39.551 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:39.551 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:39.551 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:39.545+0000 7f58c9984080 0 bucket checkpoint complete 2020-11-29T16:06:39.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:39.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:39.663 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:39.709 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:39.709 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:39.959 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:39.957+0000 7ff38cd7f080 1 bucket sync caught up with source: 2020-11-29T16:06:39.959 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.71.5, , ] 2020-11-29T16:06:39.960 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.71.5, , ] 2020-11-29T16:06:39.960 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:39.957+0000 7ff38cd7f080 0 bucket checkpoint complete 2020-11-29T16:06:39.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:39.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:39.988 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:40.076 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:40.077 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:40.301 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:40.297+0000 7f6e7a9ac080 1 bucket sync caught up with source: 2020-11-29T16:06:40.301 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.71.5, , ] 2020-11-29T16:06:40.301 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.71.5, , ] 2020-11-29T16:06:40.302 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:40.297+0000 7f6e7a9ac080 0 bucket checkpoint complete 2020-11-29T16:06:40.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:40.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:40.336 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:40.383 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:40.383 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:40.642 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:40.641+0000 7f22ed140080 1 bucket sync caught up with source: 2020-11-29T16:06:40.642 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.73.5, , ] 2020-11-29T16:06:40.643 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.73.5, , ] 2020-11-29T16:06:40.643 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:06:40.641+0000 7f22ed140080 0 bucket checkpoint complete 2020-11-29T16:06:40.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:40.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:40.669 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:40.721 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:40.721 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:40.973 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:40.969+0000 7f05a3afd080 1 bucket sync caught up with source: 2020-11-29T16:06:40.974 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.73.5, , ] 2020-11-29T16:06:40.975 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.73.5, , ] 2020-11-29T16:06:40.975 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:40.969+0000 7f05a3afd080 0 bucket checkpoint complete 2020-11-29T16:06:41.008 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone2, test-zone1} 2020-11-29T16:06:41.021 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:06:41.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:41.071 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:06:41.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:41.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone2, test-zone1} 2020-11-29T16:06:41.275 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone3, test-zone1} 2020-11-29T16:06:41.285 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:06:41.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:41.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:06:41.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:41.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.533 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone3, test-zone1} 2020-11-29T16:06:41.533 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone3, test-zone2} 2020-11-29T16:06:41.545 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:06:41.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:41.591 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:06:41.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:06:41.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:06:41.813 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone3, test-zone2} 2020-11-29T16:06:41.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:41.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:41.836 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:41.878 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:41.879 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:42.110 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:06:42.143 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:06:42.144 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:42.144 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:06:42.144 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:06:42.144 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:42.145 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:42.145 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666001.830749_100.1", 2020-11-29T16:06:42.145 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:06:41.830749Z" 2020-11-29T16:06:42.145 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:42.145 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:42.146 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:06:42.146 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:42.146 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:42.146 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:42.147 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:06:42.147 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:06:42.147 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:42.147 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:06:42.148 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:06:42.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:42.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:42.150 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:42.190 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:42.190 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:42.452 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:06:42.452 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:06:42.453 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:06:42.453 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:06:42.453 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:06:42.453 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:06:42.453 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:42.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:42.454 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:06:42.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:42.454 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:06:42.455 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:42.455 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:42.455 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:06:42.455 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:42.455 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:06:42.456 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:42.456 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:06:42.456 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:42.456 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:42.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:42.457 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:42.457 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:06:42.457 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:42.457 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:42.457 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666001.830749_100.1", 2020-11-29T16:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:41.830749Z", 2020-11-29T16:06:42.458 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:42.459 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:06:42.460 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:42.460 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:42.460 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:06:42.460 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:42.460 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:42.461 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:42.461 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:06:42.461 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:42.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:42.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:42.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:42.462 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:06:42.462 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:42.462 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:42.462 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:06:42.463 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:42.463 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:42.463 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:42.463 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:06:42.463 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:06:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:42.464 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:06:42.465 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:06:42.465 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:06:42.465 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:42.465 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:06:42.466 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:06:42.468 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:06:42.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:42.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:42.471 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:42.512 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:42.512 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:42.760 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:06:42.761 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:06:42.761 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:06:42.762 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:06:42.762 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:06:42.762 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:06:42.763 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:42.763 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:42.764 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:06:42.764 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:42.764 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:06:42.765 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:42.765 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:06:42.766 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:06:42.766 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:42.767 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:06:42.767 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:42.767 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:06:42.768 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:42.768 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:42.769 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:42.769 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:42.769 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:06:42.770 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:42.770 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:06:42.771 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666001.830749_100.1", 2020-11-29T16:06:42.771 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:42.771 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:42.772 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:42.772 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:41.830749Z", 2020-11-29T16:06:42.773 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:42.773 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:42.773 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:42.774 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:42.774 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:06:42.775 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:42.775 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:06:42.775 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:06:42.776 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:42.776 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:42.776 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:42.777 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:06:42.777 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:42.778 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:42.778 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:42.778 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:06:42.779 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:06:42.779 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:06:42.780 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:06:42.780 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:06:42.780 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:06:42.781 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:06:42.781 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:06:42.782 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:06:42.782 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:06:42.782 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:42.783 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:42.783 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:06:42.784 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:06:42.784 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:06:42.784 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:06:42.785 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:06:42.785 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:06:42.786 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:06:42.788 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:06:42.789 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:06:42.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:42.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:06:42.791 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:06:42.834 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:42.835 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:43.099 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:06:43.099 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:06:43.099 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:06:43.099 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:06:43.100 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:06:43.100 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:06:43.100 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:43.100 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:43.101 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:06:43.101 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:43.101 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:06:43.101 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:43.101 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:43.102 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:06:43.102 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:43.102 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:06:43.102 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:43.102 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:06:43.103 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:43.103 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:43.103 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:43.103 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:43.104 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:06:43.104 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:43.104 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:43.104 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666001.830749_100.1", 2020-11-29T16:06:43.104 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:43.105 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:43.105 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:43.105 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:41.830749Z", 2020-11-29T16:06:43.105 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:43.106 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:43.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:43.106 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:43.106 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:06:43.106 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:43.107 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:43.107 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:06:43.107 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:43.107 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:43.108 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:43.108 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:06:43.108 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:43.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:43.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:43.109 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:06:43.109 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:06:43.109 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:06:43.109 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:06:43.109 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:06:43.110 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:06:43.110 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:06:43.110 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:06:43.110 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:06:43.110 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:06:43.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:43.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:43.111 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:06:43.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:06:43.112 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:06:43.112 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:06:43.112 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:06:43.112 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:06:43.112 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:06:43.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:06:43.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:43.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:06:43.164 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:06:43.211 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:06:43.212 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:06:43.446 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:43.441+0000 7f78d3684080 1 waiting for incremental sync to catch up: 2020-11-29T16:06:43.447 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:43.447 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:06:48.450 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:48.445+0000 7f78d3684080 1 waiting for incremental sync to catch up: 2020-11-29T16:06:48.451 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:48.451 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:06:53.455 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:53.449+0000 7f78d3684080 1 waiting for incremental sync to catch up: 2020-11-29T16:06:53.455 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:53.455 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:06:58.459 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:06:58.453+0000 7f78d3684080 1 waiting for incremental sync to catch up: 2020-11-29T16:06:58.459 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.49.5, , ] 2020-11-29T16:06:58.460 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:07:03.463 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:03.457+0000 7f78d3684080 1 bucket sync caught up with source: 2020-11-29T16:07:03.463 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:07:03.464 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:07:03.464 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:03.457+0000 7f78d3684080 0 bucket checkpoint complete 2020-11-29T16:07:03.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:03.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:03.496 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:03.542 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:03.543 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:03.798 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:03.796+0000 7fc82b5d4080 1 bucket sync caught up with source: 2020-11-29T16:07:03.798 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:07:03.798 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:07:03.799 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:03.796+0000 7fc82b5d4080 0 bucket checkpoint complete 2020-11-29T16:07:03.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:03.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:03.826 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:03.882 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:04.106 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:04.101+0000 7f0750558080 1 bucket sync caught up with source: 2020-11-29T16:07:04.106 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:07:04.107 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.52.12, , ] 2020-11-29T16:07:04.107 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:04.101+0000 7f0750558080 0 bucket checkpoint complete 2020-11-29T16:07:04.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:04.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:04.238 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:04.286 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:04.287 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:04.525 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:04.520+0000 7f3a0aaed080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:04.525 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.71.5, , ] 2020-11-29T16:07:04.525 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.77.12, , ] 2020-11-29T16:07:09.533 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:09.528+0000 7f3a0aaed080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:09.533 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.71.5, , ] 2020-11-29T16:07:09.533 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.77.12, , ] 2020-11-29T16:07:14.537 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:14.532+0000 7f3a0aaed080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:14.537 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.71.5, , ] 2020-11-29T16:07:14.538 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.77.12, , ] 2020-11-29T16:07:19.541 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:19.536+0000 7f3a0aaed080 1 bucket sync caught up with source: 2020-11-29T16:07:19.542 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000004.79.9, , ] 2020-11-29T16:07:19.542 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.77.12, , ] 2020-11-29T16:07:19.542 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:19.536+0000 7f3a0aaed080 0 bucket checkpoint complete 2020-11-29T16:07:19.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:19.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:19.576 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:19.620 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:19.621 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:19.876 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:19.872+0000 7fdc5cdfe080 1 bucket sync caught up with source: 2020-11-29T16:07:19.876 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000004.79.9, , ] 2020-11-29T16:07:19.877 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000004.79.9, , ] 2020-11-29T16:07:19.877 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:19.872+0000 7fdc5cdfe080 0 bucket checkpoint complete 2020-11-29T16:07:19.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:19.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:19.908 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:20.002 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:20.002 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:20.252 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:20.248+0000 7f07bf3bf080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:20.252 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.79.12, , ] 2020-11-29T16:07:20.252 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000004.81.9, , ] 2020-11-29T16:07:25.256 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:25.252+0000 7f07bf3bf080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:25.257 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.79.12, , ] 2020-11-29T16:07:25.257 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000004.81.9, , ] 2020-11-29T16:07:30.260 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:30.256+0000 7f07bf3bf080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:30.260 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.79.12, , ] 2020-11-29T16:07:30.261 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000004.81.9, , ] 2020-11-29T16:07:35.263 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:35.260+0000 7f07bf3bf080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:35.264 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.79.12, , ] 2020-11-29T16:07:35.264 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000004.81.9, , ] 2020-11-29T16:07:40.268 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:40.264+0000 7f07bf3bf080 1 bucket sync caught up with source: 2020-11-29T16:07:40.269 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000004.81.9, , ] 2020-11-29T16:07:40.269 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000004.81.9, , ] 2020-11-29T16:07:40.269 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:40.264+0000 7f07bf3bf080 0 bucket checkpoint complete 2020-11-29T16:07:40.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:40.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:40.302 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:40.350 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:40.350 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:40.585 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:40.584+0000 7fe1592ae080 1 bucket sync caught up with source: 2020-11-29T16:07:40.585 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000004.81.9, , ] 2020-11-29T16:07:40.585 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000004.81.9, , ] 2020-11-29T16:07:40.585 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:40.584+0000 7fe1592ae080 0 bucket checkpoint complete 2020-11-29T16:07:40.630 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone2, test-zone1} 2020-11-29T16:07:40.641 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:07:40.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:40.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:40.692 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:07:40.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:40.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:40.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:40.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:40.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:40.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:40.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:40.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.020 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone2, test-zone1} 2020-11-29T16:07:41.020 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone3, test-zone1} 2020-11-29T16:07:41.028 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:07:41.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:41.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:41.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:07:41.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:41.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:41.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.396 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone3, test-zone1} 2020-11-29T16:07:41.397 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone3, test-zone2} 2020-11-29T16:07:41.408 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:07:41.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:41.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:41.455 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:07:41.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:41.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:07:41.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:07:41.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone3, test-zone2} 2020-11-29T16:07:41.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:07:41.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:07:41.825 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:07:41.866 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:41.867 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:42.058 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:07:42.091 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:07:42.092 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:07:42.092 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:07:42.092 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:07:42.093 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:07:42.093 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:07:42.093 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666061.819824_107.1", 2020-11-29T16:07:42.093 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:07:41.819824Z" 2020-11-29T16:07:42.093 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:07:42.094 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:07:42.094 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:07:42.094 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:07:42.095 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:07:42.095 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:07:42.095 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:07:42.095 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:07:42.095 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:07:42.096 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:07:42.096 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:07:42.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:07:42.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:07:42.098 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:07:42.138 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:42.138 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:42.406 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:07:42.406 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:07:42.406 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:07:42.407 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:07:42.407 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:07:42.407 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:07:42.407 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:42.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:42.408 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:07:42.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:07:42.408 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:07:42.409 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:07:42.409 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:07:42.409 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:07:42.409 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:07:42.410 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:07:42.410 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:07:42.410 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:07:42.410 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:42.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:42.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:42.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:07:42.411 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:07:42.411 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:07:42.412 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:07:42.412 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666061.819824_107.1", 2020-11-29T16:07:42.412 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:07:42.412 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:07:42.413 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:07:42.413 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:07:41.819824Z", 2020-11-29T16:07:42.413 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:42.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:42.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:42.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:07:42.414 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:07:42.414 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:07:42.414 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:07:42.415 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:07:42.415 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:07:42.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:07:42.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:07:42.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:07:42.416 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:42.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:42.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:42.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:07:42.417 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:07:42.417 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:07:42.417 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:07:42.417 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:07:42.418 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:07:42.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:07:42.418 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:07:42.419 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:07:42.419 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:42.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:42.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:42.420 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:07:42.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:42.420 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:07:42.420 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:07:42.420 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:07:42.421 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:42.421 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:07:42.422 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:07:42.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:07:42.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:07:42.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:07:42.424 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:07:42.463 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:42.464 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:42.707 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:07:42.708 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:07:42.708 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:07:42.708 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:07:42.708 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:07:42.709 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:07:42.709 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:07:42.709 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:07:42.709 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:07:42.709 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:07:42.710 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:07:42.710 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:07:42.710 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:07:42.710 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:07:42.711 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:07:42.711 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:07:42.711 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:07:42.711 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:07:42.711 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:07:42.712 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:07:42.712 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:07:42.712 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:07:42.712 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:07:42.712 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:07:42.713 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:07:42.713 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666061.819824_107.1", 2020-11-29T16:07:42.713 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:07:42.713 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:07:42.713 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:07:42.714 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:07:41.819824Z", 2020-11-29T16:07:42.714 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:07:42.714 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:07:42.714 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:07:42.714 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:07:42.715 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:07:42.715 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:07:42.715 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:07:42.715 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:07:42.715 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:07:42.716 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:07:42.716 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:07:42.716 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:07:42.716 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:07:42.717 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:07:42.717 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:07:42.717 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:07:42.717 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:07:42.717 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:07:42.718 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:07:42.718 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:07:42.718 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:07:42.718 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:07:42.718 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:07:42.719 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:07:42.719 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:07:42.719 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:07:42.719 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:07:42.719 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:07:42.720 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:07:42.720 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:07:42.720 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:07:42.720 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:07:42.720 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:07:42.721 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:07:42.722 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:07:42.722 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:07:42.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:07:42.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:07:42.724 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:07:42.767 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:42.768 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:43.035 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:07:43.036 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:07:43.036 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:07:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:07:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:07:43.037 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:07:43.038 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:43.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:43.039 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:07:43.039 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:07:43.039 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:07:43.040 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:07:43.040 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:07:43.041 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:07:43.041 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:07:43.042 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:07:43.042 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:07:43.042 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:07:43.043 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:43.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:07:43.044 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:07:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:07:43.045 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:07:43.046 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666061.819824_107.1", 2020-11-29T16:07:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:07:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:07:43.047 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:07:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:07:41.819824Z", 2020-11-29T16:07:43.048 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:43.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:07:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:07:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:07:43.050 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:07:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:07:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:07:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:07:43.051 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:07:43.052 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:07:43.052 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:43.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:43.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:43.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:07:43.053 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:07:43.053 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:07:43.053 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:07:43.053 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:07:43.054 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:07:43.054 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:07:43.054 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:07:43.054 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:07:43.055 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:07:43.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:43.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:43.055 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:07:43.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:07:43.056 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:07:43.056 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:07:43.056 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:07:43.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:07:43.057 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:07:43.058 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:07:43.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:43.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:43.112 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:43.380 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:43.380+0000 7f98ebaa6080 1 bucket sync caught up with source: 2020-11-29T16:07:43.380 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000005.56.17, , ] 2020-11-29T16:07:43.381 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.56.17, , ] 2020-11-29T16:07:43.381 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:43.380+0000 7f98ebaa6080 0 bucket checkpoint complete 2020-11-29T16:07:43.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:43.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:43.418 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:43.466 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:43.480 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:43.718 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:43.716+0000 7f2c60f74080 1 bucket sync caught up with source: 2020-11-29T16:07:43.718 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000005.56.17, , ] 2020-11-29T16:07:43.718 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000005.56.17, , ] 2020-11-29T16:07:43.719 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:43.716+0000 7f2c60f74080 0 bucket checkpoint complete 2020-11-29T16:07:43.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:43.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:43.747 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:43.795 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:43.796 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:44.034 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:44.032+0000 7f7efd662080 1 bucket sync caught up with source: 2020-11-29T16:07:44.035 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000005.56.17, , ] 2020-11-29T16:07:44.035 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.56.17, , ] 2020-11-29T16:07:44.035 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:07:44.032+0000 7f7efd662080 0 bucket checkpoint complete 2020-11-29T16:07:44.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:44.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:44.168 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:44.216 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:44.216 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:44.446 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:44.444+0000 7fc838d34080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:44.446 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000004.79.9, , ] 2020-11-29T16:07:44.446 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000006.86.17, , ] 2020-11-29T16:07:49.450 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:49.448+0000 7fc838d34080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:49.450 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000004.79.9, , ] 2020-11-29T16:07:49.451 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000006.86.17, , ] 2020-11-29T16:07:54.454 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:54.451+0000 7fc838d34080 1 waiting for incremental sync to catch up: 2020-11-29T16:07:54.454 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000004.79.9, , ] 2020-11-29T16:07:54.454 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000006.86.17, , ] 2020-11-29T16:07:59.457 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:59.451+0000 7fc838d34080 1 bucket sync caught up with source: 2020-11-29T16:07:59.458 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000007.88.14, , ] 2020-11-29T16:07:59.458 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000006.86.17, , ] 2020-11-29T16:07:59.458 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:59.451+0000 7fc838d34080 0 bucket checkpoint complete 2020-11-29T16:07:59.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:59.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:59.493 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:59.539 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:59.540 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:07:59.787 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:59.783+0000 7f416fb5a080 1 bucket sync caught up with source: 2020-11-29T16:07:59.787 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000007.88.14, , ] 2020-11-29T16:07:59.788 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000007.88.14, , ] 2020-11-29T16:07:59.788 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:07:59.783+0000 7f416fb5a080 0 bucket checkpoint complete 2020-11-29T16:07:59.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:59.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:07:59.815 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:07:59.861 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:07:59.862 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:00.084 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:00.079+0000 7fc6b7ae0080 1 waiting for incremental sync to catch up: 2020-11-29T16:08:00.084 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000006.88.17, , ] 2020-11-29T16:08:00.084 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000007.90.14, , ] 2020-11-29T16:08:05.088 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:05.083+0000 7fc6b7ae0080 1 waiting for incremental sync to catch up: 2020-11-29T16:08:05.089 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000006.88.17, , ] 2020-11-29T16:08:05.089 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000007.90.14, , ] 2020-11-29T16:08:10.092 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:10.087+0000 7fc6b7ae0080 1 waiting for incremental sync to catch up: 2020-11-29T16:08:10.092 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000006.88.17, , ] 2020-11-29T16:08:10.092 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000007.90.14, , ] 2020-11-29T16:08:15.095 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:15.091+0000 7fc6b7ae0080 1 waiting for incremental sync to catch up: 2020-11-29T16:08:15.096 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000006.88.17, , ] 2020-11-29T16:08:15.096 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000007.90.14, , ] 2020-11-29T16:08:20.100 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:20.095+0000 7fc6b7ae0080 1 bucket sync caught up with source: 2020-11-29T16:08:20.100 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000007.90.14, , ] 2020-11-29T16:08:20.101 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000007.90.14, , ] 2020-11-29T16:08:20.101 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:20.095+0000 7fc6b7ae0080 0 bucket checkpoint complete 2020-11-29T16:08:20.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:20.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:20.132 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:20.180 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:20.181 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:20.426 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:20.423+0000 7f7ea1f56080 1 bucket sync caught up with source: 2020-11-29T16:08:20.427 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000007.90.14, , ] 2020-11-29T16:08:20.427 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000007.90.14, , ] 2020-11-29T16:08:20.427 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:20.423+0000 7f7ea1f56080 0 bucket checkpoint complete 2020-11-29T16:08:20.459 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone2, test-zone1} 2020-11-29T16:08:20.471 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:20.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:20.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:20.524 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:20.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:20.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:20.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:20.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:20.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:20.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:20.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:20.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:20.886 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone2, test-zone1} 2020-11-29T16:08:20.887 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone3, test-zone1} 2020-11-29T16:08:20.896 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:20.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:20.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:20.943 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:20.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:20.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:21.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.283 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone3, test-zone1} 2020-11-29T16:08:21.283 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-20 zones={test-zone3, test-zone2} 2020-11-29T16:08:21.296 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:21.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:21.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:21.347 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:21.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:21.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:21.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:21.710 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-20 zones={test-zone3, test-zone2} 2020-11-29T16:08:21.711 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-11-29T16:08:21.712 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-21 2020-11-29T16:08:21.751 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-22 2020-11-29T16:08:21.812 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-23 2020-11-29T16:08:21.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:21.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:21.935 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:21.976 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:21.976 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:22.181 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:08:22.211 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:08:22.211 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:22.211 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:22.211 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:08:22.212 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:22.212 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:22.212 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666101.915755_125.1", 2020-11-29T16:08:22.212 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:21.915755Z" 2020-11-29T16:08:22.213 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:22.213 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:22.213 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:22.213 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:08:22.213 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:22.214 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:22.214 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:22.214 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:08:22.214 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:22.215 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:08:22.216 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:08:22.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:22.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:22.217 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:22.254 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:22.254 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:22.507 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:22.508 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:22.508 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:22.508 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:22.508 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:22.508 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:22.509 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:22.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:22.509 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:22.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:22.509 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:22.510 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:22.510 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:22.510 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:22.510 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:22.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:22.511 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:22.511 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:22.511 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:22.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:22.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:22.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:22.512 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:22.512 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:22.512 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:22.513 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666101.915755_125.1", 2020-11-29T16:08:22.513 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:22.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:22.513 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:22.513 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.915755Z", 2020-11-29T16:08:22.514 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:22.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:22.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:22.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:22.515 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:22.515 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:22.515 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:22.515 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:22.515 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:22.516 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:22.516 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:22.516 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:22.516 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:22.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:22.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:22.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:22.517 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:22.517 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:22.517 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:22.518 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:22.518 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:22.518 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:22.518 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:22.518 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:22.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:22.520 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:22.520 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:22.520 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:22.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:22.520 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:22.522 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:08:22.522 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:08:22.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:22.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:22.524 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:22.565 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:22.565 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:22.839 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:08:22.839 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:08:22.840 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:08:22.840 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:08:22.840 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:08:22.841 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:22.841 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:22.842 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:22.842 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:08:22.842 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:22.843 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:08:22.843 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:22.844 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:22.844 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:22.845 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:22.845 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:08:22.845 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:22.846 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:22.846 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:22.846 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:22.847 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:22.847 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:22.847 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:08:22.847 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:22.848 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:22.848 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666101.915755_125.1", 2020-11-29T16:08:22.848 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:22.849 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:22.849 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:22.849 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.915755Z", 2020-11-29T16:08:22.849 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:22.850 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:22.850 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:22.850 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:22.850 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:08:22.851 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:22.851 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:22.851 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:22.851 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:22.851 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:22.852 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:22.852 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:08:22.852 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:22.852 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:22.852 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:22.853 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:22.853 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:08:22.853 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:22.853 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:22.853 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:22.854 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:22.854 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:22.854 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:22.854 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:08:22.855 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:22.855 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:22.855 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:22.855 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:08:22.855 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:22.856 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:08:22.856 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:08:22.856 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:08:22.857 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:22.857 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:08:22.858 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:08:22.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:08:22.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:22.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:22.861 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:22.900 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:22.901 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:23.165 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:23.165 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:23.166 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:23.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:23.166 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:23.166 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:23.167 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:23.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:23.167 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:23.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:23.168 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:23.168 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:23.168 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:23.169 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:23.169 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:23.170 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:23.170 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:23.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:23.171 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:23.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:23.173 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:23.174 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666101.915755_125.1", 2020-11-29T16:08:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:23.175 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.915755Z", 2020-11-29T16:08:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:23.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:23.177 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:23.178 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:23.179 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:23.180 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:23.180 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:23.180 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:23.180 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:23.180 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:23.181 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:23.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:23.183 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:23.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:23.184 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:23.185 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:08:23.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:23.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:23.411 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:23.461 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:23.462 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:23.683 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:23.679+0000 7f3f0bbc7080 1 bucket sync caught up with source: 2020-11-29T16:08:23.683 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.8.11, , ] 2020-11-29T16:08:23.683 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.8.11, , ] 2020-11-29T16:08:23.683 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:23.679+0000 7f3f0bbc7080 0 bucket checkpoint complete 2020-11-29T16:08:23.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:23.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:23.711 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:23.762 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:23.762 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:23.998 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:23.995+0000 7fc359262080 1 bucket sync caught up with source: 2020-11-29T16:08:23.998 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.8.11, , ] 2020-11-29T16:08:23.999 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.8.11, , ] 2020-11-29T16:08:23.999 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:23.995+0000 7fc359262080 0 bucket checkpoint complete 2020-11-29T16:08:24.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:24.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:24.024 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:24.072 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:24.073 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:24.320 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:24.319+0000 7fcae0e97080 1 bucket sync caught up with source: 2020-11-29T16:08:24.321 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.8.11, , ] 2020-11-29T16:08:24.321 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.8.11, , ] 2020-11-29T16:08:24.321 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:24.319+0000 7fcae0e97080 0 bucket checkpoint complete 2020-11-29T16:08:24.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:24.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:24.444 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:24.493 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:24.494 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:24.738 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:24.735+0000 7f47594b5080 1 bucket sync caught up with source: 2020-11-29T16:08:24.739 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2020-11-29T16:08:24.739 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2020-11-29T16:08:24.739 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:24.735+0000 7f47594b5080 0 bucket checkpoint complete 2020-11-29T16:08:24.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:24.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:24.772 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:24.818 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:24.818 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:25.052 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:25.047+0000 7f168c19f080 1 bucket sync caught up with source: 2020-11-29T16:08:25.053 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2020-11-29T16:08:25.053 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2020-11-29T16:08:25.053 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:25.047+0000 7f168c19f080 0 bucket checkpoint complete 2020-11-29T16:08:25.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:25.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:25.086 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:25.134 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:25.134 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:25.373 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:25.367+0000 7f1b4a412080 1 bucket sync caught up with source: 2020-11-29T16:08:25.373 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2020-11-29T16:08:25.373 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2020-11-29T16:08:25.374 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:25.367+0000 7f1b4a412080 0 bucket checkpoint complete 2020-11-29T16:08:25.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:25.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:25.403 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:25.450 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:25.451 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:25.716 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:25.715+0000 7f7479f2a080 1 bucket sync caught up with source: 2020-11-29T16:08:25.716 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2020-11-29T16:08:25.716 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2020-11-29T16:08:25.717 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:25.715+0000 7f7479f2a080 0 bucket checkpoint complete 2020-11-29T16:08:25.748 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-21 zones={test-zone2, test-zone1} 2020-11-29T16:08:25.759 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:25.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:25.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:25.807 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:25.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:25.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:25.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:25.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:25.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:25.964 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-21 zones={test-zone2, test-zone1} 2020-11-29T16:08:25.965 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-21 zones={test-zone3, test-zone1} 2020-11-29T16:08:25.972 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:26.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:26.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:26.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:26.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:26.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:26.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:26.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:26.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:26.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-21 zones={test-zone3, test-zone1} 2020-11-29T16:08:26.173 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-21 zones={test-zone3, test-zone2} 2020-11-29T16:08:26.182 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:26.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:26.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:26.231 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:26.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:26.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:26.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:26.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:26.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:26.380 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-21 zones={test-zone3, test-zone2} 2020-11-29T16:08:26.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:26.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:26.382 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:26.427 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:26.427 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:26.665 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:26.663+0000 7f3797d2b080 1 bucket sync caught up with source: 2020-11-29T16:08:26.666 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.119.5, , ] 2020-11-29T16:08:26.666 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.119.5, , ] 2020-11-29T16:08:26.666 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:26.663+0000 7f3797d2b080 0 bucket checkpoint complete 2020-11-29T16:08:26.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:26.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:26.698 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:26.746 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:26.746 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:26.982 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:26.979+0000 7f1d345b7080 1 bucket sync caught up with source: 2020-11-29T16:08:26.983 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000002.119.5, , ] 2020-11-29T16:08:26.983 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000002.119.5, , ] 2020-11-29T16:08:26.983 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:26.979+0000 7f1d345b7080 0 bucket checkpoint complete 2020-11-29T16:08:27.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:27.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:27.011 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:27.061 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:27.061 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:27.326 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:27.323+0000 7f476c8fc080 1 bucket sync caught up with source: 2020-11-29T16:08:27.326 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.119.5, , ] 2020-11-29T16:08:27.327 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.119.5, , ] 2020-11-29T16:08:27.327 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:27.323+0000 7f476c8fc080 0 bucket checkpoint complete 2020-11-29T16:08:27.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:27.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:27.438 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:27.486 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:27.486 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:27.723 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:27.719+0000 7fee11d83080 1 bucket sync caught up with source: 2020-11-29T16:08:27.723 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.132.11, , ] 2020-11-29T16:08:27.724 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.132.11, , ] 2020-11-29T16:08:27.724 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:27.719+0000 7fee11d83080 0 bucket checkpoint complete 2020-11-29T16:08:27.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:27.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:27.755 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:27.844 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:27.845 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:28.085 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:28.079+0000 7fc2748e8080 1 bucket sync caught up with source: 2020-11-29T16:08:28.085 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.132.11, , ] 2020-11-29T16:08:28.086 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.132.11, , ] 2020-11-29T16:08:28.086 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:28.079+0000 7fc2748e8080 0 bucket checkpoint complete 2020-11-29T16:08:28.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:28.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:28.119 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:28.164 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:28.164 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:28.383 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:28.379+0000 7f629512d080 1 bucket sync caught up with source: 2020-11-29T16:08:28.384 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000002.131.5, , ] 2020-11-29T16:08:28.384 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000002.131.5, , ] 2020-11-29T16:08:28.384 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:28.379+0000 7f629512d080 0 bucket checkpoint complete 2020-11-29T16:08:28.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:28.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:28.412 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:28.460 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:28.460 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:28.724 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:28.719+0000 7f59598cb080 1 bucket sync caught up with source: 2020-11-29T16:08:28.724 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.131.5, , ] 2020-11-29T16:08:28.724 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.131.5, , ] 2020-11-29T16:08:28.725 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:28.719+0000 7f59598cb080 0 bucket checkpoint complete 2020-11-29T16:08:28.756 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-22 zones={test-zone2, test-zone1} 2020-11-29T16:08:28.768 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:28.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:28.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:28.815 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:28.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:28.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:28.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:28.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:28.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:28.964 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-22 zones={test-zone2, test-zone1} 2020-11-29T16:08:28.964 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-22 zones={test-zone3, test-zone1} 2020-11-29T16:08:28.973 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:29.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:29.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:29.019 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:29.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:29.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:29.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:29.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:29.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:29.168 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-22 zones={test-zone3, test-zone1} 2020-11-29T16:08:29.169 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-22 zones={test-zone3, test-zone2} 2020-11-29T16:08:29.177 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:29.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:29.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:29.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:29.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:29.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:29.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:29.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:29.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:29.384 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-22 zones={test-zone3, test-zone2} 2020-11-29T16:08:29.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:29.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:29.386 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:29.430 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:29.430 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:29.677 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:29.675+0000 7f94c53cd080 1 bucket sync caught up with source: 2020-11-29T16:08:29.678 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2020-11-29T16:08:29.679 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2020-11-29T16:08:29.679 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:29.675+0000 7f94c53cd080 0 bucket checkpoint complete 2020-11-29T16:08:29.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:29.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:29.715 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:29.764 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:29.764 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:30.009 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:30.003+0000 7f719f116080 1 bucket sync caught up with source: 2020-11-29T16:08:30.010 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2020-11-29T16:08:30.010 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2020-11-29T16:08:30.010 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:30.003+0000 7f719f116080 0 bucket checkpoint complete 2020-11-29T16:08:30.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:30.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:30.041 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:30.089 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:30.090 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:30.335 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:30.331+0000 7f2918233080 1 bucket sync caught up with source: 2020-11-29T16:08:30.335 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.10.5, , ] 2020-11-29T16:08:30.335 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.10.5, , ] 2020-11-29T16:08:30.336 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:30.331+0000 7f2918233080 0 bucket checkpoint complete 2020-11-29T16:08:30.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:30.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:30.465 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:30.513 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:30.514 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:30.776 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:30.771+0000 7f48357df080 1 bucket sync caught up with source: 2020-11-29T16:08:30.777 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2020-11-29T16:08:30.777 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2020-11-29T16:08:30.777 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:30.771+0000 7f48357df080 0 bucket checkpoint complete 2020-11-29T16:08:30.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:30.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:30.809 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:30.855 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:30.855 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:31.096 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:31.091+0000 7f70ee8cd080 1 bucket sync caught up with source: 2020-11-29T16:08:31.097 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000002.16.5, , ] 2020-11-29T16:08:31.097 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000002.16.5, , ] 2020-11-29T16:08:31.097 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:31.091+0000 7f70ee8cd080 0 bucket checkpoint complete 2020-11-29T16:08:31.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:31.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:31.134 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:31.179 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:31.180 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:31.415 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:31.411+0000 7f1ce1ebe080 1 bucket sync caught up with source: 2020-11-29T16:08:31.416 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.17.11, , ] 2020-11-29T16:08:31.416 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.17.11, , ] 2020-11-29T16:08:31.416 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:31.411+0000 7f1ce1ebe080 0 bucket checkpoint complete 2020-11-29T16:08:31.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:31.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:31.446 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:31.494 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:31.494 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:31.745 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:31.743+0000 7f7d5bd72080 1 bucket sync caught up with source: 2020-11-29T16:08:31.745 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.17.11, , ] 2020-11-29T16:08:31.746 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.17.11, , ] 2020-11-29T16:08:31.746 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:31.743+0000 7f7d5bd72080 0 bucket checkpoint complete 2020-11-29T16:08:31.776 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-23 zones={test-zone2, test-zone1} 2020-11-29T16:08:31.788 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:31.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:31.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:31.839 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:31.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:31.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:31.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:31.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:31.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:31.998 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-23 zones={test-zone2, test-zone1} 2020-11-29T16:08:31.998 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-23 zones={test-zone3, test-zone1} 2020-11-29T16:08:32.008 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:32.058 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:32.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:32.059 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:32.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:32.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:32.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:32.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:32.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:32.217 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-23 zones={test-zone3, test-zone1} 2020-11-29T16:08:32.217 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-23 zones={test-zone3, test-zone2} 2020-11-29T16:08:32.227 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:32.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:32.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:32.275 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:32.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:32.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:32.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:32.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:32.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:32.437 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-23 zones={test-zone3, test-zone2} 2020-11-29T16:08:32.438 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-11-29T16:08:32.438 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-24 2020-11-29T16:08:32.472 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-25 2020-11-29T16:08:32.536 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-26 2020-11-29T16:08:32.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:32.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:32.693 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:32.736 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:32.737 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:32.939 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:08:32.972 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:08:32.972 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:32.973 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:32.973 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:08:32.973 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:32.973 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:32.973 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666112.684234_149.1", 2020-11-29T16:08:32.974 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:32.684234Z" 2020-11-29T16:08:32.974 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:32.974 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:32.974 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:32.975 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:08:32.975 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:32.975 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:32.976 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:32.976 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:08:32.976 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:32.976 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:08:32.977 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:08:32.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:32.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:32.979 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:33.029 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:33.029 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:33.279 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:33.279 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:33.279 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:33.280 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:33.280 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:33.280 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:33.280 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.281 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:33.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:33.281 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:33.281 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:33.281 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:33.282 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:33.282 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:33.282 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:33.282 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:33.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666112.684234_149.1", 2020-11-29T16:08:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.684234Z", 2020-11-29T16:08:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.292 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:33.293 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:08:33.293 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:08:33.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:33.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:33.294 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:33.333 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:33.334 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:33.597 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:08:33.597 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:08:33.597 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:08:33.598 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:08:33.598 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:08:33.598 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:33.598 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:33.598 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:33.599 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:08:33.599 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:33.599 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:08:33.599 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:33.600 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:33.600 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:33.600 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:33.600 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:08:33.601 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:33.601 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:33.601 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:33.601 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:33.601 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:33.602 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:33.602 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:08:33.602 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:33.602 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:33.603 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666112.684234_149.1", 2020-11-29T16:08:33.603 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:33.603 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:33.603 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:33.603 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.684234Z", 2020-11-29T16:08:33.604 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:33.604 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:33.604 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:33.604 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:33.605 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:08:33.605 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:33.605 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:33.605 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:33.605 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:33.606 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:33.606 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:33.606 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:08:33.606 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:33.607 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:33.607 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:33.607 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:33.607 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:08:33.607 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:33.608 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:33.608 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:33.608 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:33.608 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:33.608 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:33.609 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:08:33.609 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:33.609 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:33.609 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:33.609 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:08:33.610 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:33.610 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:08:33.610 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:08:33.610 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:08:33.611 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:33.611 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:08:33.612 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:08:33.612 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:08:33.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:33.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:33.614 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:33.655 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:33.656 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:33.930 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:33.932 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:33.932 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:33.936 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:33.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:33.939 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:33.939 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:33.940 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:33.940 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666112.684234_149.1", 2020-11-29T16:08:33.941 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:33.941 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:33.941 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:33.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.684234Z", 2020-11-29T16:08:33.942 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:33.944 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:33.945 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:33.945 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:33.945 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:33.946 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:33.946 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:33.947 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:33.947 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:33.948 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:33.949 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:33.949 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:33.950 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:33.950 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:33.950 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:33.950 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:33.951 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:33.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.952 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:33.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:33.953 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:33.953 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:33.953 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:33.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:33.954 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:33.955 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:08:34.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:34.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:34.196 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:34.245 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:34.246 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:34.473 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:34.471+0000 7f240ced0080 1 bucket sync caught up with source: 2020-11-29T16:08:34.473 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.66.6, , ] 2020-11-29T16:08:34.473 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.66.6, , ] 2020-11-29T16:08:34.474 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:34.471+0000 7f240ced0080 0 bucket checkpoint complete 2020-11-29T16:08:34.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:34.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:34.502 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:34.548 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:34.549 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:34.756 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:34.751+0000 7f920d3d6080 1 bucket sync caught up with source: 2020-11-29T16:08:34.757 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.66.6, , ] 2020-11-29T16:08:34.757 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.66.6, , ] 2020-11-29T16:08:34.757 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:34.751+0000 7f920d3d6080 0 bucket checkpoint complete 2020-11-29T16:08:34.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:34.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:34.789 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:34.836 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:34.837 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:35.074 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:35.071+0000 7f5d5fd1b080 1 bucket sync caught up with source: 2020-11-29T16:08:35.074 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.66.6, , ] 2020-11-29T16:08:35.075 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.66.6, , ] 2020-11-29T16:08:35.075 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:35.071+0000 7f5d5fd1b080 0 bucket checkpoint complete 2020-11-29T16:08:35.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:35.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:35.108 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:35.154 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:35.155 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:35.393 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:35.387+0000 7fa05f466080 1 bucket sync caught up with source: 2020-11-29T16:08:35.393 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.100.12, , ] 2020-11-29T16:08:35.393 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.100.12, , ] 2020-11-29T16:08:35.394 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:35.387+0000 7fa05f466080 0 bucket checkpoint complete 2020-11-29T16:08:35.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:35.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:35.428 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:35.475 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:35.476 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:35.720 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:35.715+0000 7f47d9189080 1 bucket sync caught up with source: 2020-11-29T16:08:35.721 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.100.12, , ] 2020-11-29T16:08:35.721 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.100.12, , ] 2020-11-29T16:08:35.721 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:35.715+0000 7f47d9189080 0 bucket checkpoint complete 2020-11-29T16:08:35.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:35.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:35.756 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:35.802 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:35.802 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:36.037 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:36.031+0000 7f9af48ce080 1 bucket sync caught up with source: 2020-11-29T16:08:36.037 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.96.12, , ] 2020-11-29T16:08:36.037 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.96.12, , ] 2020-11-29T16:08:36.038 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:36.031+0000 7f9af48ce080 0 bucket checkpoint complete 2020-11-29T16:08:36.064 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:36.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:36.065 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:36.114 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:36.114 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:36.362 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:36.359+0000 7f791e46f080 1 bucket sync caught up with source: 2020-11-29T16:08:36.362 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.96.12, , ] 2020-11-29T16:08:36.363 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.96.12, , ] 2020-11-29T16:08:36.363 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:36.359+0000 7f791e46f080 0 bucket checkpoint complete 2020-11-29T16:08:36.408 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-24 zones={test-zone2, test-zone1} 2020-11-29T16:08:36.419 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:36.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:36.471 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:36.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:36.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:36.542 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-24 zones={test-zone2, test-zone1} 2020-11-29T16:08:36.543 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-24 zones={test-zone3, test-zone1} 2020-11-29T16:08:36.551 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:36.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:36.599 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:36.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:36.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:36.665 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-24 zones={test-zone3, test-zone1} 2020-11-29T16:08:36.665 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-24 zones={test-zone3, test-zone2} 2020-11-29T16:08:36.675 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:36.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:36.723 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:36.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:36.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:36.794 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-24 zones={test-zone3, test-zone2} 2020-11-29T16:08:36.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:36.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:36.796 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:36.841 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:36.842 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:37.093 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:37.091+0000 7f7fcce3b080 1 bucket sync caught up with source: 2020-11-29T16:08:37.094 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.20.12, , ] 2020-11-29T16:08:37.094 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.20.12, , ] 2020-11-29T16:08:37.094 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:37.091+0000 7f7fcce3b080 0 bucket checkpoint complete 2020-11-29T16:08:37.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:37.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:37.126 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:37.174 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:37.174 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:37.405 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:37.398+0000 7f4326322080 1 bucket sync caught up with source: 2020-11-29T16:08:37.405 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.20.12, , ] 2020-11-29T16:08:37.406 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.20.12, , ] 2020-11-29T16:08:37.406 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:37.398+0000 7f4326322080 0 bucket checkpoint complete 2020-11-29T16:08:37.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:37.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:37.434 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:37.480 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:37.481 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:37.735 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:37.731+0000 7f72b08fb080 1 bucket sync caught up with source: 2020-11-29T16:08:37.735 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.20.12, , ] 2020-11-29T16:08:37.735 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.20.12, , ] 2020-11-29T16:08:37.736 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:37.731+0000 7f72b08fb080 0 bucket checkpoint complete 2020-11-29T16:08:37.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:37.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:37.852 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:37.901 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:37.902 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:38.150 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:38.146+0000 7f8c0059a080 1 bucket sync caught up with source: 2020-11-29T16:08:38.151 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.34.6, , ] 2020-11-29T16:08:38.151 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.34.6, , ] 2020-11-29T16:08:38.151 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:38.146+0000 7f8c0059a080 0 bucket checkpoint complete 2020-11-29T16:08:38.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:38.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:38.184 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:38.230 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:38.231 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:38.446 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:38.442+0000 7f0babf65080 1 bucket sync caught up with source: 2020-11-29T16:08:38.446 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.34.6, , ] 2020-11-29T16:08:38.447 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.34.6, , ] 2020-11-29T16:08:38.447 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:38.442+0000 7f0babf65080 0 bucket checkpoint complete 2020-11-29T16:08:38.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:38.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:38.478 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:38.565 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:38.565 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:38.819 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:38.814+0000 7f166b22d080 1 bucket sync caught up with source: 2020-11-29T16:08:38.820 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.29.12, , ] 2020-11-29T16:08:38.820 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.29.12, , ] 2020-11-29T16:08:38.820 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:38.814+0000 7f166b22d080 0 bucket checkpoint complete 2020-11-29T16:08:38.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:38.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:38.848 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:38.897 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:38.897 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:39.149 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:39.146+0000 7f3bdda78080 1 bucket sync caught up with source: 2020-11-29T16:08:39.149 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.29.12, , ] 2020-11-29T16:08:39.150 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.29.12, , ] 2020-11-29T16:08:39.150 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:39.146+0000 7f3bdda78080 0 bucket checkpoint complete 2020-11-29T16:08:39.183 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-25 zones={test-zone2, test-zone1} 2020-11-29T16:08:39.194 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:39.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:39.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:39.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:39.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:39.311 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-25 zones={test-zone2, test-zone1} 2020-11-29T16:08:39.312 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-25 zones={test-zone3, test-zone1} 2020-11-29T16:08:39.320 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:39.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:39.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:39.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:39.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:39.442 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-25 zones={test-zone3, test-zone1} 2020-11-29T16:08:39.442 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-25 zones={test-zone3, test-zone2} 2020-11-29T16:08:39.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:39.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:39.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:39.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:39.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:39.571 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-25 zones={test-zone3, test-zone2} 2020-11-29T16:08:39.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:39.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:39.573 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:39.620 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:39.620 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:39.868 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:39.866+0000 7fb1dda68080 1 bucket sync caught up with source: 2020-11-29T16:08:39.868 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2020-11-29T16:08:39.868 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2020-11-29T16:08:39.869 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:39.866+0000 7fb1dda68080 0 bucket checkpoint complete 2020-11-29T16:08:39.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:39.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:39.902 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:39.951 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:39.952 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:40.185 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:40.178+0000 7f8b3d450080 1 bucket sync caught up with source: 2020-11-29T16:08:40.185 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2020-11-29T16:08:40.185 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2020-11-29T16:08:40.185 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:40.178+0000 7f8b3d450080 0 bucket checkpoint complete 2020-11-29T16:08:40.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:40.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:40.212 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:40.260 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:40.261 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:40.499 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:40.494+0000 7fdefb856080 1 bucket sync caught up with source: 2020-11-29T16:08:40.499 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2020-11-29T16:08:40.499 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2020-11-29T16:08:40.500 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:40.494+0000 7fdefb856080 0 bucket checkpoint complete 2020-11-29T16:08:40.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:40.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:40.626 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:40.682 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:40.682 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:40.904 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:40.898+0000 7f2aa451e080 1 bucket sync caught up with source: 2020-11-29T16:08:40.904 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.19.12, , ] 2020-11-29T16:08:40.905 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.19.12, , ] 2020-11-29T16:08:40.905 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:40.898+0000 7f2aa451e080 0 bucket checkpoint complete 2020-11-29T16:08:40.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:40.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:40.937 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:40.985 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:40.985 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:41.227 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:41.222+0000 7fafda2c3080 1 bucket sync caught up with source: 2020-11-29T16:08:41.227 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000001.19.12, , ] 2020-11-29T16:08:41.228 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000001.19.12, , ] 2020-11-29T16:08:41.228 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:41.222+0000 7fafda2c3080 0 bucket checkpoint complete 2020-11-29T16:08:41.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:41.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:41.261 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:41.353 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:41.353 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:41.582 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:41.574+0000 7f809fcd3080 1 bucket sync caught up with source: 2020-11-29T16:08:41.582 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , , , , 00000000003.29.6, , ] 2020-11-29T16:08:41.582 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , , , , 00000000003.29.6, , ] 2020-11-29T16:08:41.583 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:08:41.574+0000 7f809fcd3080 0 bucket checkpoint complete 2020-11-29T16:08:41.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:41.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:08:41.609 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:08:41.658 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:41.659 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:41.890 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:41.886+0000 7fca26a10080 1 bucket sync caught up with source: 2020-11-29T16:08:41.890 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.29.6, , ] 2020-11-29T16:08:41.891 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.29.6, , ] 2020-11-29T16:08:41.891 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:08:41.886+0000 7fca26a10080 0 bucket checkpoint complete 2020-11-29T16:08:41.920 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-26 zones={test-zone2, test-zone1} 2020-11-29T16:08:41.931 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:41.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:41.987 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:41.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:42.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:42.059 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-26 zones={test-zone2, test-zone1} 2020-11-29T16:08:42.060 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-26 zones={test-zone3, test-zone1} 2020-11-29T16:08:42.067 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:42.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:42.115 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:42.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:42.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:42.186 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-26 zones={test-zone3, test-zone1} 2020-11-29T16:08:42.186 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-26 zones={test-zone3, test-zone2} 2020-11-29T16:08:42.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:08:42.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:42.247 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:08:42.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-11-29T16:08:42.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-11-29T16:08:42.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-26 zones={test-zone3, test-zone2} 2020-11-29T16:08:42.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-11-29T16:08:42.319 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-27 2020-11-29T16:08:42.361 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-28 2020-11-29T16:08:42.437 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-29 2020-11-29T16:08:42.686 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-11-29T16:08:42.687 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-30 2020-11-29T16:08:42.731 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-31 2020-11-29T16:08:42.803 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-32 2020-11-29T16:08:43.054 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-11-29T16:08:43.055 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-33 2020-11-29T16:08:43.094 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-34 2020-11-29T16:08:43.198 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-35 2020-11-29T16:08:45.458 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-11-29T16:08:45.459 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-36 2020-11-29T16:08:45.497 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-37 2020-11-29T16:08:45.567 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-38 2020-11-29T16:08:45.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:45.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:45.624 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:45.659 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:45.660 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:45.878 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:08:45.911 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:08:45.911 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:45.911 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:45.912 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:08:45.912 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:45.912 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:45.912 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666125.602582_219.1", 2020-11-29T16:08:45.912 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:45.602582Z" 2020-11-29T16:08:45.913 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:45.913 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:45.913 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:45.913 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:08:45.914 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:45.914 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:45.914 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:45.914 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:08:45.915 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:45.915 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:08:45.916 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:08:45.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:45.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:45.917 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:45.959 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:45.960 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:46.209 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:46.209 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:46.210 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:46.210 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:46.210 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:46.210 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:46.210 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.211 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:46.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:46.211 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:46.212 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:46.212 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:46.212 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:46.212 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:46.212 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:46.213 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:46.213 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:46.213 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:46.214 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:46.214 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:46.214 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:46.215 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666125.602582_219.1", 2020-11-29T16:08:46.215 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:46.215 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:46.215 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:46.215 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.602582Z", 2020-11-29T16:08:46.216 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.216 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:46.216 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:46.217 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:46.217 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:46.217 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:46.217 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:46.217 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:46.218 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:46.218 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:46.218 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.218 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:46.219 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:46.219 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:46.219 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:46.219 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:46.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:46.220 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:46.220 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:46.220 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:46.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.221 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:46.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.222 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:46.222 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:46.222 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:46.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.222 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:46.223 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:08:46.224 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:08:46.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:46.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:46.225 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:46.265 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:46.266 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:46.518 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:08:46.518 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:08:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:08:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:08:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:08:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:46.519 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:08:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:08:46.520 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:46.521 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:46.521 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:46.521 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:46.521 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:08:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:46.522 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:08:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:46.523 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666125.602582_219.1", 2020-11-29T16:08:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.602582Z", 2020-11-29T16:08:46.524 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:46.525 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:46.525 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:46.525 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:46.525 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:08:46.526 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:46.526 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:46.526 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:46.526 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:46.526 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:46.527 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:46.527 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:08:46.527 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:46.527 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:46.527 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:46.528 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:46.528 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:08:46.528 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:46.528 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:46.528 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:46.529 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:46.529 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:46.529 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:46.529 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:08:46.529 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:46.530 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:46.530 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:46.530 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:08:46.530 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:46.531 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:08:46.531 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:08:46.531 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:08:46.531 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:46.531 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:08:46.532 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:08:46.533 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:08:46.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:46.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:46.535 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:46.576 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:46.576 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:46.945 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:46.945 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:46.946 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:46.946 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:46.946 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:46.946 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:46.946 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.947 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:46.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:46.947 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:46.948 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:46.948 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:46.948 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:46.948 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:46.948 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:46.950 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:46.950 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:46.950 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:46.950 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666125.602582_219.1", 2020-11-29T16:08:46.951 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:46.951 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:46.951 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:46.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.602582Z", 2020-11-29T16:08:46.951 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:46.952 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:46.952 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:46.953 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:46.953 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:46.953 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:46.953 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:46.953 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:46.954 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:46.954 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:46.955 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:46.955 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:46.955 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:46.955 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:46.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:46.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:46.956 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:46.956 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:46.956 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:46.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.959 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:46.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:46.960 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:46.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:46.960 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:46.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:46.961 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:46.961 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:08:47.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-11-29T16:08:47.320 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-39 2020-11-29T16:08:47.347 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-40 2020-11-29T16:08:47.405 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-41 2020-11-29T16:08:47.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:47.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:47.489 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:47.528 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:47.528 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:47.753 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=e24cddf9-396c-4261-a7bc-b4e1e678325e 2020-11-29T16:08:47.786 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:08:47.786 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:47.787 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:47.787 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T15:59:59.895400Z" 2020-11-29T16:08:47.787 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:47.787 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:47.787 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666127.452875_235.1", 2020-11-29T16:08:47.788 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:47.452875Z" 2020-11-29T16:08:47.788 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:47.788 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:47.788 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:47.788 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:08:47.789 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:47.789 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:47.789 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:47.789 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:08:47.789 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:47.790 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:08:47.790 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:08:47.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:47.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:47.792 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:47.832 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:47.833 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:48.085 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:48.086 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:48.086 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:48.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:48.086 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:48.087 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:48.087 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:48.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:48.087 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:48.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:48.088 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:48.088 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:48.088 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:48.088 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:48.088 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:48.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:48.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:48.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:48.090 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:48.090 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:48.090 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:48.091 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666127.440999_232.1", 2020-11-29T16:08:48.091 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:48.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:48.091 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:48.091 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.440999Z", 2020-11-29T16:08:48.092 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:48.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:48.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:48.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:48.092 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:48.093 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:48.093 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:48.093 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:48.093 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:48.093 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:48.094 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:48.094 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:48.094 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:48.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:48.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:48.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:48.095 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:48.095 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:48.095 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:48.095 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:48.096 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:48.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:48.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:48.096 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:48.096 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:48.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:48.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666127.452875_235.1 target=1_1606666127.440999_232.1 2020-11-29T16:08:53.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:53.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:53.103 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:53.145 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:53.145 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:53.410 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:53.410 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:53.410 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:53.411 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:53.411 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:53.411 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:53.412 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:53.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:53.412 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:53.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:53.412 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:53.413 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:53.413 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:53.413 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:53.414 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:53.414 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:53.414 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:53.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:53.415 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:53.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:53.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:53.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:53.416 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:53.416 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:53.416 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:53.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666127.452875_235.1", 2020-11-29T16:08:53.417 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:53.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:53.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:53.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.452875Z", 2020-11-29T16:08:53.417 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:53.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:53.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:53.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:53.419 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:53.419 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:53.419 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:53.419 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:53.419 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:53.420 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:53.420 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:53.420 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:53.420 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:53.421 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:53.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:53.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:53.421 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:53.422 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:53.422 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:53.422 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:53.422 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:53.423 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:53.423 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:53.423 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:53.424 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:53.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:53.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:53.424 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:53.425 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:53.426 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:53.427 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:08:53.427 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:08:53.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:53.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:53.429 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:53.470 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:53.471 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:53.726 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:08:53.726 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:08:53.727 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:08:53.727 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:08:53.727 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:08:53.727 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:53.728 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:53.728 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:53.728 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:08:53.728 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:53.728 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:08:53.729 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:53.729 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:53.729 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:53.729 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:53.729 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:08:53.730 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:53.730 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:53.730 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:53.730 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:53.731 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:53.731 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:53.731 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:08:53.731 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:53.731 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:53.732 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666127.452875_235.1", 2020-11-29T16:08:53.732 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:53.732 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:53.732 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:53.732 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.452875Z", 2020-11-29T16:08:53.733 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:53.733 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:53.733 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:53.733 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:53.734 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:08:53.734 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:53.734 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:53.734 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:53.734 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:53.735 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:53.735 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:53.735 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:08:53.735 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:53.735 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:53.736 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:53.736 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:08:53.736 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:08:53.736 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:08:53.736 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:08:53.737 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:08:53.737 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:08:53.737 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:08:53.737 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:08:53.737 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:08:53.738 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:08:53.738 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:53.738 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:53.738 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:08:53.739 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:08:53.739 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:08:53.739 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:08:53.739 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:08:53.739 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:08:53.740 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:08:53.741 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:08:53.742 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:08:53.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:53.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:08:53.744 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:08:53.782 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:08:53.783 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:08:54.152 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:08:54.153 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:08:54.154 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:08:54.154 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:08:54.154 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:08:54.155 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:08:54.155 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:54.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:54.156 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:08:54.157 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:54.157 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:08:54.157 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:54.158 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:54.158 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:08:54.159 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:54.159 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:08:54.159 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:54.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:08:54.160 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:54.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:54.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:54.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:54.162 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:08:54.162 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:54.163 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:54.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666127.452875_235.1", 2020-11-29T16:08:54.164 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:54.164 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:54.164 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:54.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.452875Z", 2020-11-29T16:08:54.165 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:54.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:54.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:54.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:54.167 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:08:54.167 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:54.168 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:54.168 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:54.168 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:54.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:54.169 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:54.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:54.170 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:54.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:54.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:54.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:08:54.172 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:08:54.172 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:08:54.173 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:08:54.173 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:08:54.173 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:08:54.174 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:08:54.174 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:08:54.175 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:08:54.175 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:08:54.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:54.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:54.176 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:08:54.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:08:54.177 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:08:54.178 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:08:54.178 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:08:54.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:08:54.179 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:08:54.181 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:08:54.182 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-11-29T16:08:54.183 INFO:teuthology.orchestra.run:waiting for 300 2020-11-29T16:08:54.184 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:08:54.178+0000 7f631cf04700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18033) UID: 0 2020-11-29T16:08:54.185 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:08:54.178+0000 7f63349b70c0 -1 shutting down 2020-11-29T16:08:56.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-11-29T16:09:00.285 INFO:tasks.rgw.c1.client.1:Stopped 2020-11-29T16:09:00.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:09:00.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:09:00.288 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:09:00.330 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:09:00.331 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:09:00.361 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:09:00.358+0000 7fd777855080 0 NOTICE: overriding master zone: a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T16:09:00.362 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:09:00.358+0000 7fd777855080 0 failed reading obj info from .rgw.root:zone_info.36679b47-164f-436b-b728-5846b32cea58: (2) No such file or directory 2020-11-29T16:09:00.362 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:09:00.358+0000 7fd777855080 0 WARNING: could not read zone params for zone id=36679b47-164f-436b-b728-5846b32cea58 name=test-zone3 2020-11-29T16:09:00.364 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:09:00.362+0000 7fd777855080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T16:09:00.364 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:09:00.362+0000 7fd777855080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T16:09:00.375 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:09:00.375 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:09:00.376 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:09:00.376 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-29T16:09:00.376 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-29T16:09:00.377 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-29T16:09:00.377 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-29T16:09:00.377 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-29T16:09:00.377 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-29T16:09:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-29T16:09:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-29T16:09:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-29T16:09:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-29T16:09:00.378 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-29T16:09:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-29T16:09:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-29T16:09:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-29T16:09:00.379 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T16:09:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T16:09:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T16:09:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T16:09:00.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T16:09:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:09:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-29T16:09:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T16:09:00.381 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T16:09:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-29T16:09:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T16:09:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-29T16:09:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T16:09:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.383 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.384 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:09:00.384 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-29T16:09:00.384 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:09:00.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:09:00.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:09:00.386 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:09:00.465 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:09:00.466 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:09:00.772 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:09:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:09:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 1, 2020-11-29T16:09:00.773 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "e24cddf9-396c-4261-a7bc-b4e1e678325e", 2020-11-29T16:09:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-11-29T16:09:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1606665599.895400_96.1", 2020-11-29T16:09:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1606666127.452875_235.1", 2020-11-29T16:09:00.774 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:09:00.775 INFO:teuthology.orchestra.run.smithi184.stdout: "" 2020-11-29T16:09:00.775 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.775 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-11-29T16:09:00.776 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:09:00.776 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-11-29T16:09:00.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.777 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-11-29T16:09:00.777 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-11-29T16:09:00.777 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-11-29T16:09:00.778 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-11-29T16:09:00.778 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:09:00.778 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:09:00.779 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.779 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-11-29T16:09:00.780 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-11-29T16:09:00.780 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:09:00.781 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-11-29T16:09:00.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.781 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:09:00.781 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-11-29T16:09:00.781 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:09:00.782 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:09:00.782 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.782 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:09:00.782 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:09:00.783 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:09:00.783 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:09:00.783 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:09:00.783 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:09:00.783 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:09:00.784 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:09:00.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.784 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:09:00.785 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:09:00.785 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:09:00.785 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:09:00.785 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.785 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:09:00.786 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:09:00.786 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:09:00.786 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:09:00.786 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:09:00.787 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:09:00.787 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:09:00.787 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:09:00.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.788 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.788 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:09:00.788 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T16:09:00.788 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:09:00.788 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:09:00.789 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.789 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:09:00.789 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:09:00.789 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:09:00.790 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:09:00.790 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-11-29T16:09:00.790 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T16:09:00.790 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-11-29T16:09:00.791 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-11-29T16:09:00.791 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.791 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:09:00.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.792 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:09:00.792 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-11-29T16:09:00.792 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:09:00.793 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:09:00.793 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.793 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:09:00.793 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:09:00.793 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:09:00.794 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:09:00.794 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:09:00.794 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:09:00.794 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:09:00.795 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:09:00.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.795 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.795 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-11-29T16:09:00.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.796 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-11-29T16:09:00.796 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-11-29T16:09:00.796 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-11-29T16:09:00.796 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-11-29T16:09:00.796 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:09:00.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.797 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.797 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-11-29T16:09:00.798 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:09:00.798 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-11-29T16:09:00.798 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-11-29T16:09:00.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.799 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:09:00.800 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-11-29T16:09:00.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.801 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:09:00.801 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 791374490 2020-11-29T16:09:00.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.802 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:09:00.802 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2491292681 2020-11-29T16:09:00.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.803 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:09:00.803 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 293306859 2020-11-29T16:09:00.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:09:00.804 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:09:00.804 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1785095687 2020-11-29T16:09:00.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.804 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:09:00.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.805 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:09:00.805 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:09:00.805 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-11-29T16:09:00.805 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-11-29T16:09:00.805 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:09:00.806 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:09:00.806 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:09:00.806 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:09:00.806 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:09:00.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.807 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-11-29T16:09:00.807 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:09:00.807 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:09:00.807 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:09:00.808 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:09:00.808 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:09:00.808 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:09:00.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:09:00.809 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:09:00.809 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-11-29T16:09:00.809 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 3 2020-11-29T16:09:00.809 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:09:00.810 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-11-29T16:09:02.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-11-29T16:10:00.811 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-42 2020-11-29T16:10:00.887 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-43 2020-11-29T16:10:00.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:10:00.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:10:00.936 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:10:00.976 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:10:00.976 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:10:01.206 INFO:teuthology.orchestra.run.smithi184.stderr:No --period given, using current period=f053a404-d473-420f-9a01-38eedc856256 2020-11-29T16:10:01.239 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:10:01.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:10:01.240 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:10:01.240 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-11-29T16:10:01.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:10:01.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:10:01.241 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666200.930041_148.1", 2020-11-29T16:10:01.241 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:10:00.930041Z" 2020-11-29T16:10:01.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:10:01.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:10:01.242 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:10:01.242 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-11-29T16:10:01.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:10:01.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:10:01.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:10:01.243 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-11-29T16:10:01.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:10:01.244 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:10:01.245 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-11-29T16:10:01.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:10:01.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:10:01.246 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:10:01.291 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:10:01.293 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:10:01.539 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:10:01.540 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:10:01.540 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:10:01.541 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:10:01.541 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:10:01.542 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:10:01.542 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 3 2020-11-29T16:10:01.542 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:01.543 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:10:01.543 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:01.544 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:10:01.544 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:10:01.544 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:10:01.545 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:10:01.545 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:10:01.546 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 80, 2020-11-29T16:10:01.546 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:10:01.546 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:10:01.547 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 3 2020-11-29T16:10:01.547 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.548 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:01.548 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:01.548 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:10:01.549 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:10:01.549 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:10:01.550 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666200.930041_148.1", 2020-11-29T16:10:01.550 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:10:01.550 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:10:01.551 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:10:01.551 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.930041Z", 2020-11-29T16:10:01.552 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 3 2020-11-29T16:10:01.552 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.552 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:01.553 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:01.553 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:10:01.554 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:10:01.554 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:10:01.554 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:10:01.555 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:10:01.555 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:10:01.556 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:10:01.556 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:10:01.556 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 3 2020-11-29T16:10:01.557 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.557 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:01.558 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:01.558 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:10:01.558 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:10:01.559 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:10:01.559 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:10:01.560 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:10:01.560 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:10:01.560 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:10:01.561 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:10:01.561 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 3 2020-11-29T16:10:01.562 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.562 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.562 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:10:01.563 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:01.563 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:10:01.564 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 80, 2020-11-29T16:10:01.564 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 80 2020-11-29T16:10:01.564 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.565 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:10:01.566 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-11-29T16:10:01.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:10:01.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:10:01.569 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:10:01.650 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:10:01.650 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:10:01.680 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:10:01.677+0000 7f41a99a1080 0 NOTICE: overriding master zone: 3c165286-3738-4e0f-b399-acd859302998 2020-11-29T16:10:01.681 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:10:01.677+0000 7f41a99a1080 0 failed reading obj info from .rgw.root:zone_info.3c165286-3738-4e0f-b399-acd859302998: (2) No such file or directory 2020-11-29T16:10:01.681 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:10:01.677+0000 7f41a99a1080 0 WARNING: could not read zone params for zone id=3c165286-3738-4e0f-b399-acd859302998 name=test-zone2 2020-11-29T16:10:01.682 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:10:01.677+0000 7f41a99a1080 0 failed reading obj info from .rgw.root:zone_info.82b01792-db79-401e-8298-c419db80f2a2: (2) No such file or directory 2020-11-29T16:10:01.683 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:10:01.677+0000 7f41a99a1080 0 WARNING: could not read zone params for zone id=82b01792-db79-401e-8298-c419db80f2a2 name=test-zone4 2020-11-29T16:10:01.694 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:10:01.694 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:10:01.695 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T16:10:01.695 INFO:teuthology.orchestra.run.smithi168.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-11-29T16:10:01.695 INFO:teuthology.orchestra.run.smithi168.stdout: "control_pool": "test-zone1.rgw.control", 2020-11-29T16:10:01.696 INFO:teuthology.orchestra.run.smithi168.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-11-29T16:10:01.696 INFO:teuthology.orchestra.run.smithi168.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-11-29T16:10:01.697 INFO:teuthology.orchestra.run.smithi168.stdout: "log_pool": "test-zone1.rgw.log", 2020-11-29T16:10:01.697 INFO:teuthology.orchestra.run.smithi168.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-11-29T16:10:01.698 INFO:teuthology.orchestra.run.smithi168.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-11-29T16:10:01.698 INFO:teuthology.orchestra.run.smithi168.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-11-29T16:10:01.699 INFO:teuthology.orchestra.run.smithi168.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-11-29T16:10:01.699 INFO:teuthology.orchestra.run.smithi168.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-11-29T16:10:01.700 INFO:teuthology.orchestra.run.smithi168.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-11-29T16:10:01.700 INFO:teuthology.orchestra.run.smithi168.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-11-29T16:10:01.700 INFO:teuthology.orchestra.run.smithi168.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-11-29T16:10:01.701 INFO:teuthology.orchestra.run.smithi168.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-11-29T16:10:01.701 INFO:teuthology.orchestra.run.smithi168.stdout: "system_key": { 2020-11-29T16:10:01.702 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T16:10:01.702 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T16:10:01.702 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:01.703 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_pools": [ 2020-11-29T16:10:01.703 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:01.704 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "default-placement", 2020-11-29T16:10:01.704 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:10:01.704 INFO:teuthology.orchestra.run.smithi168.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-11-29T16:10:01.705 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": { 2020-11-29T16:10:01.705 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD": { 2020-11-29T16:10:01.706 INFO:teuthology.orchestra.run.smithi168.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-11-29T16:10:01.706 INFO:teuthology.orchestra.run.smithi168.stdout: "compression_type": "random" 2020-11-29T16:10:01.706 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.707 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:01.707 INFO:teuthology.orchestra.run.smithi168.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-11-29T16:10:01.708 INFO:teuthology.orchestra.run.smithi168.stdout: "index_type": 0 2020-11-29T16:10:01.708 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.708 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:01.708 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:01.709 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:10:01.709 INFO:teuthology.orchestra.run.smithi168.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-11-29T16:10:01.709 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:10:01.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:10:01.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:10:01.712 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:10:01.805 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:10:01.805 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:10:02.117 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:10:02.118 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:10:02.118 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 1, 2020-11-29T16:10:02.118 INFO:teuthology.orchestra.run.smithi168.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:10:02.119 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": [ 2020-11-29T16:10:02.119 INFO:teuthology.orchestra.run.smithi168.stdout: "", 2020-11-29T16:10:02.119 INFO:teuthology.orchestra.run.smithi168.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:10:02.119 INFO:teuthology.orchestra.run.smithi168.stdout: "", 2020-11-29T16:10:02.119 INFO:teuthology.orchestra.run.smithi168.stdout: "" 2020-11-29T16:10:02.120 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.120 INFO:teuthology.orchestra.run.smithi168.stdout: "period_map": { 2020-11-29T16:10:02.120 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:10:02.120 INFO:teuthology.orchestra.run.smithi168.stdout: "zonegroups": [ 2020-11-29T16:10:02.121 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.121 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "test-zonegroup", 2020-11-29T16:10:02.121 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zonegroup", 2020-11-29T16:10:02.121 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "", 2020-11-29T16:10:02.121 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "true", 2020-11-29T16:10:02.122 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:10:02.122 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:10:02.122 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.122 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T16:10:02.122 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T16:10:02.123 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:10:02.123 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [ 2020-11-29T16:10:02.123 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.123 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:10:02.123 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone3", 2020-11-29T16:10:02.124 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:10:02.124 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:10:02.124 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.124 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:10:02.124 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:10:02.125 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:10:02.125 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:10:02.125 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:10:02.125 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:10:02.126 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:10:02.126 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:10:02.126 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.126 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.126 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:10:02.127 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone2", 2020-11-29T16:10:02.127 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:10:02.127 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:10:02.127 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.127 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:10:02.128 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:10:02.128 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:10:02.128 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:10:02.128 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:10:02.128 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:10:02.129 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:10:02.129 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:10:02.129 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.129 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.129 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:10:02.130 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone4", 2020-11-29T16:10:02.130 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:10:02.130 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:10:02.130 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.131 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:10:02.131 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:10:02.131 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:10:02.131 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:10:02.131 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "pubsub", 2020-11-29T16:10:02.132 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "false", 2020-11-29T16:10:02.132 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [ 2020-11-29T16:10:02.132 INFO:teuthology.orchestra.run.smithi168.stdout: "test-zone1" 2020-11-29T16:10:02.132 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.132 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:10:02.133 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.133 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.133 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:10:02.133 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T16:10:02.133 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:10:02.134 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:10:02.134 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.134 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:10:02.134 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:10:02.134 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:10:02.135 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:10:02.135 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:10:02.135 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:10:02.135 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:10:02.135 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:10:02.136 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:02.136 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.136 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [ 2020-11-29T16:10:02.136 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.137 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default-placement", 2020-11-29T16:10:02.137 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-11-29T16:10:02.137 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": [ 2020-11-29T16:10:02.137 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD" 2020-11-29T16:10:02.137 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:10:02.138 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:02.138 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.138 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "default-placement", 2020-11-29T16:10:02.138 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:10:02.138 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T16:10:02.139 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T16:10:02.139 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:02.139 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:02.139 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:10:02.146 INFO:teuthology.orchestra.run.smithi168.stdout: "short_zone_ids": [ 2020-11-29T16:10:02.147 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.147 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:10:02.147 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 791374490 2020-11-29T16:10:02.147 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.148 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.148 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:10:02.148 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 2491292681 2020-11-29T16:10:02.148 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.148 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.149 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:10:02.149 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 293306859 2020-11-29T16:10:02.149 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.149 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:10:02.150 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:10:02.150 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 1785095687 2020-11-29T16:10:02.150 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:02.150 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:10:02.150 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.151 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:10:02.151 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:10:02.151 INFO:teuthology.orchestra.run.smithi168.stdout: "period_config": { 2020-11-29T16:10:02.151 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_quota": { 2020-11-29T16:10:02.151 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T16:10:02.152 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T16:10:02.152 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T16:10:02.152 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T16:10:02.152 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T16:10:02.152 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.153 INFO:teuthology.orchestra.run.smithi168.stdout: "user_quota": { 2020-11-29T16:10:02.153 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T16:10:02.153 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T16:10:02.153 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T16:10:02.153 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T16:10:02.154 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T16:10:02.154 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:10:02.154 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:10:02.154 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:10:02.154 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_name": "test-realm", 2020-11-29T16:10:02.155 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:10:02.155 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:10:02.156 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-11-29T16:11:02.157 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-44 2020-11-29T16:11:02.201 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-45 2020-11-29T16:11:02.282 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-11-29T16:11:02.283 INFO:teuthology.orchestra.run.smithi168:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-11-29T16:11:02.285 INFO:tasks.rgw.c1.client.1:Started 2020-11-29T16:11:02.286 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-11-29T16:11:02.296 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T16:11:02.296 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T16:11:02.300 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi168.front.sepia.ceph.com port 8001: Connection refused 2020-11-29T16:11:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-11-29T16:11:03.302 INFO:teuthology.orchestra.run.smithi168:> curl http://smithi168.front.sepia.ceph.com:8001 2020-11-29T16:11:03.319 INFO:teuthology.orchestra.run.smithi168.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-11-29T16:11:03.319 INFO:teuthology.orchestra.run.smithi168.stderr: Dload Upload Total Spent Left Speed 2020-11-29T16:11:03.324 INFO:teuthology.orchestra.run.smithi168.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-11-29T16:11:03.325 INFO:teuthology.orchestra.run.smithi168.stdout:anonymous 2020-11-29T16:11:03.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:03.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:03.326 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:03.409 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:03.409 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:03.614 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:11:03.651 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:03.651 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:03.652 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:03.652 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:11:03.653 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:03.653 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:03.653 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666262.250986_219.1", 2020-11-29T16:11:03.654 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:02.250986Z" 2020-11-29T16:11:03.654 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:03.655 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:03.656 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:03.656 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:11:03.657 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:03.657 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:03.658 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:03.658 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:11:03.658 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:03.659 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:03.661 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:11:03.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:03.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:03.664 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:03.713 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:03.714 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:04.004 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:04.005 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:11:04.005 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:11:04.005 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:11:04.006 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:11:04.006 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:11:04.006 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:04.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.007 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:11:04.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:04.007 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:11:04.007 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:04.007 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:04.008 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:11:04.008 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:04.008 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:11:04.008 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:04.009 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:11:04.009 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:04.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:04.010 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:11:04.010 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:04.010 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:04.010 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666262.250986_219.1", 2020-11-29T16:11:04.011 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:04.011 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:04.011 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:04.011 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.250986Z", 2020-11-29T16:11:04.012 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:04.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:04.012 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:11:04.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:04.013 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:04.013 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:11:04.013 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:04.014 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:04.014 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:04.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:11:04.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:04.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:04.015 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:11:04.015 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:04.015 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:04.016 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:11:04.016 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:04.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:04.016 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:04.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:11:04.017 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:04.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.017 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:04.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.018 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:11:04.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:11:04.018 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:11:04.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.019 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:04.020 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:11:04.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:11:04.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:04.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:04.022 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:04.064 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:04.064 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:04.317 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:11:04.317 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:11:04.318 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:11:04.318 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:11:04.318 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:11:04.319 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:11:04.319 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:11:04.319 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:04.319 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:11:04.320 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:04.320 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:11:04.320 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:04.320 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:11:04.321 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:11:04.321 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:04.321 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:11:04.321 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:04.322 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:11:04.322 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:11:04.322 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:04.322 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:04.325 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:04.326 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:11:04.326 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:04.326 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:11:04.326 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666262.250986_219.1", 2020-11-29T16:11:04.327 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:04.327 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:04.327 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:04.327 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.250986Z", 2020-11-29T16:11:04.327 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:11:04.328 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:04.328 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:04.328 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:04.328 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:11:04.328 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:04.329 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:11:04.329 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:04.329 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:04.329 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:04.329 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:04.330 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:11:04.330 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:11:04.330 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:04.330 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:04.331 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:04.331 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:11:04.331 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:04.331 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:11:04.331 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:04.332 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:04.332 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:04.332 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:04.332 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:11:04.332 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:11:04.333 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:04.333 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:04.333 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:11:04.333 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:04.333 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:11:04.334 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:11:04.334 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:11:04.334 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:04.334 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:11:04.335 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:11:04.336 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:11:04.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:04.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:04.337 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:04.378 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:04.378 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:04.744 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:04.744 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:11:04.744 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:11:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:11:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:11:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:11:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:04.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.746 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:11:04.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:04.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:11:04.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:04.747 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:04.747 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:11:04.747 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:04.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:11:04.747 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:04.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:11:04.748 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:04.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:04.749 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:11:04.749 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:04.749 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:04.749 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666262.250986_219.1", 2020-11-29T16:11:04.750 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:04.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:04.750 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:04.750 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.250986Z", 2020-11-29T16:11:04.750 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:04.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:04.751 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:11:04.751 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:04.752 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:04.752 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:11:04.752 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:04.752 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:11:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:04.754 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:11:04.754 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:04.754 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:04.755 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:11:04.755 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:04.755 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:04.755 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:04.755 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:11:04.756 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:04.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.756 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:04.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:04.757 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:11:04.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:11:04.757 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:11:04.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:04.758 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:04.759 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:11:04.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-39 zones={test-zone2, test-zone1} 2020-11-29T16:11:04.781 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:04.834 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:04.905 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-39 zones={test-zone2, test-zone1} 2020-11-29T16:11:04.905 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-39 zones={test-zone3, test-zone1} 2020-11-29T16:11:04.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:04.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:05.036 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-39 zones={test-zone3, test-zone1} 2020-11-29T16:11:05.036 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-39 zones={test-zone1, test-zone2} 2020-11-29T16:11:05.046 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:05.094 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:05.162 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-39 zones={test-zone1, test-zone2} 2020-11-29T16:11:05.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-39 zones={test-zone3, test-zone2} 2020-11-29T16:11:05.173 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:05.222 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:05.285 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-39 zones={test-zone3, test-zone2} 2020-11-29T16:11:05.286 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-39 zones={test-zone1, test-zone3} 2020-11-29T16:11:05.298 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:05.350 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:05.409 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-39 zones={test-zone1, test-zone3} 2020-11-29T16:11:05.410 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-39 zones={test-zone2, test-zone3} 2020-11-29T16:11:05.421 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:05.470 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:05.533 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-39 zones={test-zone2, test-zone3} 2020-11-29T16:11:05.534 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-40 zones={test-zone2, test-zone1} 2020-11-29T16:11:05.551 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:05.598 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:05.667 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-40 zones={test-zone2, test-zone1} 2020-11-29T16:11:05.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-40 zones={test-zone3, test-zone1} 2020-11-29T16:11:05.680 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:05.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:05.789 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-40 zones={test-zone3, test-zone1} 2020-11-29T16:11:05.790 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-40 zones={test-zone1, test-zone2} 2020-11-29T16:11:05.803 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:05.854 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:05.918 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-40 zones={test-zone1, test-zone2} 2020-11-29T16:11:05.919 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-40 zones={test-zone3, test-zone2} 2020-11-29T16:11:05.930 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:05.982 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:06.045 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-40 zones={test-zone3, test-zone2} 2020-11-29T16:11:06.046 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-40 zones={test-zone1, test-zone3} 2020-11-29T16:11:06.058 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:06.110 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:06.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-40 zones={test-zone1, test-zone3} 2020-11-29T16:11:06.173 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-40 zones={test-zone2, test-zone3} 2020-11-29T16:11:06.183 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:06.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:06.298 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-40 zones={test-zone2, test-zone3} 2020-11-29T16:11:06.299 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-41 zones={test-zone2, test-zone1} 2020-11-29T16:11:06.313 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:06.362 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:06.429 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-41 zones={test-zone2, test-zone1} 2020-11-29T16:11:06.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-41 zones={test-zone3, test-zone1} 2020-11-29T16:11:06.442 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:06.490 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:06.554 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-41 zones={test-zone3, test-zone1} 2020-11-29T16:11:06.555 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-41 zones={test-zone1, test-zone2} 2020-11-29T16:11:06.565 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:06.614 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:06.677 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-41 zones={test-zone1, test-zone2} 2020-11-29T16:11:06.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-41 zones={test-zone3, test-zone2} 2020-11-29T16:11:06.687 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:06.734 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:06.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-41 zones={test-zone3, test-zone2} 2020-11-29T16:11:06.798 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-41 zones={test-zone1, test-zone3} 2020-11-29T16:11:06.810 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:06.862 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:06.929 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-41 zones={test-zone1, test-zone3} 2020-11-29T16:11:06.929 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-41 zones={test-zone2, test-zone3} 2020-11-29T16:11:06.939 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:06.990 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:07.057 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-41 zones={test-zone2, test-zone3} 2020-11-29T16:11:07.058 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-42 zones={test-zone2, test-zone1} 2020-11-29T16:11:07.068 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:07.118 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:07.186 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-42 zones={test-zone2, test-zone1} 2020-11-29T16:11:07.187 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-42 zones={test-zone3, test-zone1} 2020-11-29T16:11:07.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:07.242 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:07.305 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-42 zones={test-zone3, test-zone1} 2020-11-29T16:11:07.305 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-42 zones={test-zone1, test-zone2} 2020-11-29T16:11:07.316 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:07.366 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:07.429 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-42 zones={test-zone1, test-zone2} 2020-11-29T16:11:07.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-42 zones={test-zone3, test-zone2} 2020-11-29T16:11:07.440 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:07.486 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:07.550 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-42 zones={test-zone3, test-zone2} 2020-11-29T16:11:07.550 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-42 zones={test-zone1, test-zone3} 2020-11-29T16:11:07.563 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:07.614 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:07.676 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-42 zones={test-zone1, test-zone3} 2020-11-29T16:11:07.676 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-42 zones={test-zone2, test-zone3} 2020-11-29T16:11:07.687 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:07.734 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:07.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-42 zones={test-zone2, test-zone3} 2020-11-29T16:11:07.797 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-43 zones={test-zone2, test-zone1} 2020-11-29T16:11:07.807 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:07.858 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:07.926 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-43 zones={test-zone2, test-zone1} 2020-11-29T16:11:07.927 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-43 zones={test-zone3, test-zone1} 2020-11-29T16:11:07.936 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:07.982 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:08.049 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-43 zones={test-zone3, test-zone1} 2020-11-29T16:11:08.049 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-43 zones={test-zone1, test-zone2} 2020-11-29T16:11:08.060 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:08.106 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:08.169 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-43 zones={test-zone1, test-zone2} 2020-11-29T16:11:08.170 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-43 zones={test-zone3, test-zone2} 2020-11-29T16:11:08.180 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:08.230 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:08.294 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-43 zones={test-zone3, test-zone2} 2020-11-29T16:11:08.295 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-43 zones={test-zone1, test-zone3} 2020-11-29T16:11:08.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:08.358 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:08.421 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-43 zones={test-zone1, test-zone3} 2020-11-29T16:11:08.422 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-43 zones={test-zone2, test-zone3} 2020-11-29T16:11:08.433 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:08.486 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:08.549 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-43 zones={test-zone2, test-zone3} 2020-11-29T16:11:08.550 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-44 zones={test-zone2, test-zone1} 2020-11-29T16:11:08.560 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:08.606 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:08.669 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-44 zones={test-zone2, test-zone1} 2020-11-29T16:11:08.670 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-44 zones={test-zone3, test-zone1} 2020-11-29T16:11:08.678 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:08.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:08.787 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-44 zones={test-zone3, test-zone1} 2020-11-29T16:11:08.788 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-44 zones={test-zone1, test-zone2} 2020-11-29T16:11:08.798 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:08.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:08.909 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-44 zones={test-zone1, test-zone2} 2020-11-29T16:11:08.909 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-44 zones={test-zone3, test-zone2} 2020-11-29T16:11:08.920 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:08.966 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:09.029 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-44 zones={test-zone3, test-zone2} 2020-11-29T16:11:09.030 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-44 zones={test-zone1, test-zone3} 2020-11-29T16:11:09.042 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:09.094 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:09.160 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-44 zones={test-zone1, test-zone3} 2020-11-29T16:11:09.161 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-44 zones={test-zone2, test-zone3} 2020-11-29T16:11:09.171 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:09.218 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:09.285 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-44 zones={test-zone2, test-zone3} 2020-11-29T16:11:09.286 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-45 zones={test-zone2, test-zone1} 2020-11-29T16:11:09.296 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:09.346 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:09.408 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-45 zones={test-zone2, test-zone1} 2020-11-29T16:11:09.409 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-45 zones={test-zone3, test-zone1} 2020-11-29T16:11:09.419 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:09.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:09.528 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-45 zones={test-zone3, test-zone1} 2020-11-29T16:11:09.529 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-45 zones={test-zone1, test-zone2} 2020-11-29T16:11:09.542 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:09.590 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:09.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-45 zones={test-zone1, test-zone2} 2020-11-29T16:11:09.655 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-45 zones={test-zone3, test-zone2} 2020-11-29T16:11:09.664 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:09.718 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:09.781 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-45 zones={test-zone3, test-zone2} 2020-11-29T16:11:09.782 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-45 zones={test-zone1, test-zone3} 2020-11-29T16:11:09.793 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:09.842 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:09.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-45 zones={test-zone1, test-zone3} 2020-11-29T16:11:09.904 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-45 zones={test-zone2, test-zone3} 2020-11-29T16:11:09.913 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:11:09.962 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:11:10.024 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-45 zones={test-zone2, test-zone3} 2020-11-29T16:11:10.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:10.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:10.025 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e24cddf9-396c-4261-a7bc-b4e1e678325e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:10.060 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:10.060 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:10.305 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:10.305 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.306 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665466.760427_46.1", 2020-11-29T16:11:10.306 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "user", 2020-11-29T16:11:10.306 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rgw-multisite-test-user", 2020-11-29T16:11:10.306 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:57:46.760427145Z", 2020-11-29T16:11:10.306 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.307 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.307 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.307 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.307 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.307 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.308 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7jrnGFazjfCMV9OgOp-5ICR", 2020-11-29T16:11:10.308 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.308 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.308 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.308 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.309 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.309 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.309 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.309 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.310 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665466.770600_48.1", 2020-11-29T16:11:10.310 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "user", 2020-11-29T16:11:10.310 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rgw-multisite-test-user", 2020-11-29T16:11:10.310 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:57:46.770600943Z", 2020-11-29T16:11:10.310 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.311 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.311 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.311 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.311 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.311 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.312 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7jrnGFazjfCMV9OgOp-5ICR", 2020-11-29T16:11:10.312 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.312 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.312 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.312 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.313 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.313 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.313 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.313 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.313 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.309764_49.1", 2020-11-29T16:11:10.314 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.314 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-1:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1", 2020-11-29T16:11:10.314 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.309764005Z", 2020-11-29T16:11:10.314 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.314 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.315 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.315 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.315 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.315 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.315 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tDx7cRFyPylOsARNJShhjdi", 2020-11-29T16:11:10.316 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.316 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.316 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.316 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.316 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.317 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.317 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.317 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.317 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.329195_52.1", 2020-11-29T16:11:10.317 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.318 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-1:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1", 2020-11-29T16:11:10.318 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.329195289Z", 2020-11-29T16:11:10.318 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.318 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.318 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.319 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.319 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.319 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.319 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tDx7cRFyPylOsARNJShhjdi", 2020-11-29T16:11:10.319 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.320 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.320 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.320 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.320 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.320 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.321 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.321 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.321 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.330857_53.1", 2020-11-29T16:11:10.321 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.321 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-1", 2020-11-29T16:11:10.322 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.330857588Z", 2020-11-29T16:11:10.322 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.322 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.322 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.322 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.323 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.323 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.323 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_da31gfzBeEr_qTz8sD5Z95W", 2020-11-29T16:11:10.323 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.323 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.324 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.324 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.324 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.324 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.324 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.325 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.325 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.334069_54.1", 2020-11-29T16:11:10.325 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.325 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-1", 2020-11-29T16:11:10.325 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.334069131Z", 2020-11-29T16:11:10.326 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.326 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.326 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.326 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.326 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.327 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.327 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_da31gfzBeEr_qTz8sD5Z95W", 2020-11-29T16:11:10.327 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.327 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.327 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.327 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.328 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.328 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.328 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.328 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.328 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.361540_55.1", 2020-11-29T16:11:10.329 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.329 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-2:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2", 2020-11-29T16:11:10.329 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.361540407Z", 2020-11-29T16:11:10.329 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.329 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.330 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.330 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.330 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.330 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.330 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jcxJvuSk8NgQdLMU8LRCoaP", 2020-11-29T16:11:10.331 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.331 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.331 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.331 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.331 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.332 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.332 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.332 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.332 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.366215_56.1", 2020-11-29T16:11:10.332 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.333 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-2:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2", 2020-11-29T16:11:10.333 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.366215003Z", 2020-11-29T16:11:10.333 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.333 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.334 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.334 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.334 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.334 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.334 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jcxJvuSk8NgQdLMU8LRCoaP", 2020-11-29T16:11:10.335 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.335 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.335 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.335 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.335 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.336 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.336 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.336 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.336 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.369096_57.1", 2020-11-29T16:11:10.336 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.337 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-2", 2020-11-29T16:11:10.337 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.369096676Z", 2020-11-29T16:11:10.337 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.337 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.337 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.337 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.338 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.338 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.338 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_F0nqKIPOuZ20dNAraH9mMFO", 2020-11-29T16:11:10.338 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.338 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.339 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.339 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.339 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.339 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.339 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.340 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.340 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.374355_58.1", 2020-11-29T16:11:10.340 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.340 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-2", 2020-11-29T16:11:10.340 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.374355442Z", 2020-11-29T16:11:10.341 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.341 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.341 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.341 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.341 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.342 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.342 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_F0nqKIPOuZ20dNAraH9mMFO", 2020-11-29T16:11:10.342 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.342 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.343 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.343 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.343 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.343 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.343 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.343 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.344 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665593.914359_59.1", 2020-11-29T16:11:10.344 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.344 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-3:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3", 2020-11-29T16:11:10.344 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.914359282Z", 2020-11-29T16:11:10.344 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.345 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.345 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.345 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.345 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.345 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.345 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_nPQIea0SAu2YbHSwS1hqdTS", 2020-11-29T16:11:10.346 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.346 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.346 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.346 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.346 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.347 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.347 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.347 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.347 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665593.919600_60.1", 2020-11-29T16:11:10.347 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.348 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-3:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3", 2020-11-29T16:11:10.348 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.919600334Z", 2020-11-29T16:11:10.348 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.348 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.348 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.349 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.349 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.349 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.349 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_nPQIea0SAu2YbHSwS1hqdTS", 2020-11-29T16:11:10.349 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.350 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.350 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.350 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.350 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.350 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.351 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.351 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.351 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665593.924502_61.1", 2020-11-29T16:11:10.351 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.351 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-3", 2020-11-29T16:11:10.351 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.924502922Z", 2020-11-29T16:11:10.352 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.352 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.352 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.352 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.352 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.353 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.353 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_hgxnPf55gFvOJXZoH0wMw6D", 2020-11-29T16:11:10.353 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.353 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.353 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.353 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.354 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.354 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.354 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.354 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.354 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665593.929811_62.1", 2020-11-29T16:11:10.355 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.355 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-3", 2020-11-29T16:11:10.355 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.929811866Z", 2020-11-29T16:11:10.355 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.355 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.356 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.356 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.356 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.356 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.356 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_hgxnPf55gFvOJXZoH0wMw6D", 2020-11-29T16:11:10.357 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.357 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.357 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.357 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.357 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.357 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.358 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.358 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.358 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.255794_63.1", 2020-11-29T16:11:10.358 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.358 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-4:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4", 2020-11-29T16:11:10.359 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.255794146Z", 2020-11-29T16:11:10.359 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.359 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.359 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.359 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.360 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.360 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.360 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_dI-T3JKYYOksP3k9JHqk_4S", 2020-11-29T16:11:10.360 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.360 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.361 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.361 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.361 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.361 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.361 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.361 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.362 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.260821_65.1", 2020-11-29T16:11:10.362 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.362 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-4:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4", 2020-11-29T16:11:10.362 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.260821437Z", 2020-11-29T16:11:10.363 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.363 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.363 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.363 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.363 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.363 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.364 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_dI-T3JKYYOksP3k9JHqk_4S", 2020-11-29T16:11:10.364 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.364 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.364 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.364 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.365 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.365 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.365 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.365 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.365 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.262726_66.1", 2020-11-29T16:11:10.366 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.366 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-4", 2020-11-29T16:11:10.366 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.262726231Z", 2020-11-29T16:11:10.366 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.366 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.367 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.367 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.367 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.369 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.370 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_H0hQB2qGLgINO4xvzkf-CBV", 2020-11-29T16:11:10.370 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.370 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.371 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.371 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.371 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.371 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.371 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.372 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.372 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.267845_67.1", 2020-11-29T16:11:10.372 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.372 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-4", 2020-11-29T16:11:10.372 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.267845461Z", 2020-11-29T16:11:10.373 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.373 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.373 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.373 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.373 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.374 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.374 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_H0hQB2qGLgINO4xvzkf-CBV", 2020-11-29T16:11:10.374 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.374 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.374 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.375 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.375 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.375 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.375 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.375 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.375 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.291993_68.1", 2020-11-29T16:11:10.376 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.376 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-5:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5", 2020-11-29T16:11:10.376 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.291993232Z", 2020-11-29T16:11:10.376 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.376 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.377 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.377 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.377 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.377 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.377 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_eNmUNSw8O2Q1UqGf3-a8M0y", 2020-11-29T16:11:10.378 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.378 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.378 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.378 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.378 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.379 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.379 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.379 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.379 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.297549_69.1", 2020-11-29T16:11:10.379 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.380 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-5:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5", 2020-11-29T16:11:10.380 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.297549197Z", 2020-11-29T16:11:10.380 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.380 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.380 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.381 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.381 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.381 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.381 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_eNmUNSw8O2Q1UqGf3-a8M0y", 2020-11-29T16:11:10.381 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.382 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.382 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.382 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.382 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.382 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.383 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.383 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.383 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.299887_70.1", 2020-11-29T16:11:10.383 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.383 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-5", 2020-11-29T16:11:10.384 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.299887736Z", 2020-11-29T16:11:10.384 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.384 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.384 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.384 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.385 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.385 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.385 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_axHo0yOC2VpaRAyEFVHuReN", 2020-11-29T16:11:10.385 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.385 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.386 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.386 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.386 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.386 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.386 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.387 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.387 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.307514_71.1", 2020-11-29T16:11:10.387 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.387 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-5", 2020-11-29T16:11:10.387 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.307514021Z", 2020-11-29T16:11:10.388 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.388 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.388 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.388 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.388 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.389 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.389 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_axHo0yOC2VpaRAyEFVHuReN", 2020-11-29T16:11:10.389 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.389 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.389 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.390 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.390 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.390 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.390 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.390 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.391 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.352257_72.1", 2020-11-29T16:11:10.391 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.391 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-6:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6", 2020-11-29T16:11:10.391 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.352257512Z", 2020-11-29T16:11:10.391 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.392 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.392 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.392 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.392 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.392 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.393 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_nbxbaUi2Y7VYrWoa72raC5j", 2020-11-29T16:11:10.393 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.393 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.393 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.393 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.393 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.394 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.394 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.394 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.394 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.359127_73.1", 2020-11-29T16:11:10.395 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.395 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-6:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6", 2020-11-29T16:11:10.395 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.359127428Z", 2020-11-29T16:11:10.395 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.395 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.395 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.396 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.396 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.396 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.396 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_nbxbaUi2Y7VYrWoa72raC5j", 2020-11-29T16:11:10.396 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.397 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.397 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.397 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.397 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.397 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.398 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.398 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.398 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.361298_74.1", 2020-11-29T16:11:10.398 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.399 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-6", 2020-11-29T16:11:10.399 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.361298888Z", 2020-11-29T16:11:10.399 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.399 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.399 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.400 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.400 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.400 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.400 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5-Sqi-mEXlqXjQFGM13_N21", 2020-11-29T16:11:10.400 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.401 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.401 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.401 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.401 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.401 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.402 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.402 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.402 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.365399_75.1", 2020-11-29T16:11:10.402 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.402 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-6", 2020-11-29T16:11:10.403 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.365399707Z", 2020-11-29T16:11:10.403 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.403 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.403 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.403 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.404 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.404 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.404 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5-Sqi-mEXlqXjQFGM13_N21", 2020-11-29T16:11:10.404 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.404 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.405 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.405 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.405 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.405 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.405 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.406 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.406 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.250797_76.1", 2020-11-29T16:11:10.406 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.406 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7", 2020-11-29T16:11:10.406 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.250797689Z", 2020-11-29T16:11:10.407 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.407 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.407 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.407 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.407 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.408 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.408 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_T6kPv-3jveF29kZeCLJlo-6", 2020-11-29T16:11:10.408 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.408 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.408 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.409 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.409 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.409 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.409 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.409 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.410 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.255494_79.1", 2020-11-29T16:11:10.410 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.410 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7", 2020-11-29T16:11:10.410 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.255494488Z", 2020-11-29T16:11:10.410 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.411 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.411 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.411 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.411 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.411 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.412 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_T6kPv-3jveF29kZeCLJlo-6", 2020-11-29T16:11:10.412 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.412 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.412 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.412 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.413 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.413 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.413 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.413 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.413 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.257909_80.1", 2020-11-29T16:11:10.414 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.414 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7", 2020-11-29T16:11:10.414 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.257909992Z", 2020-11-29T16:11:10.414 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.414 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.415 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.415 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.415 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.415 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.415 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:10.416 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.416 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.416 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.416 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.416 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.417 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.417 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.417 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.417 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.265880_81.1", 2020-11-29T16:11:10.417 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.418 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7", 2020-11-29T16:11:10.418 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.265880801Z", 2020-11-29T16:11:10.418 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.418 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.418 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.419 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.419 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.419 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.419 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:10.419 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.420 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.420 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.420 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.420 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.420 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.421 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.421 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.421 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.284752_82.1", 2020-11-29T16:11:10.421 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.421 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8", 2020-11-29T16:11:10.422 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.284752329Z", 2020-11-29T16:11:10.422 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.422 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.422 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.422 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.423 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.423 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.423 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Qg8VBYmtvIuXfPsbR8FJmF5", 2020-11-29T16:11:10.423 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.423 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.424 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.424 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.424 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.424 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.424 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.425 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.425 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.290062_83.1", 2020-11-29T16:11:10.425 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.425 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8", 2020-11-29T16:11:10.425 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.290062502Z", 2020-11-29T16:11:10.426 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.426 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.426 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.426 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.426 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.427 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.427 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Qg8VBYmtvIuXfPsbR8FJmF5", 2020-11-29T16:11:10.427 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.427 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.427 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.428 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.428 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.428 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.428 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.428 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.429 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.292190_84.1", 2020-11-29T16:11:10.429 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.429 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8", 2020-11-29T16:11:10.429 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.292190927Z", 2020-11-29T16:11:10.429 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.429 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.430 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.430 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.430 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.430 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.431 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:10.431 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.431 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.431 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.431 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.432 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.432 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.432 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.432 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.432 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.296685_85.1", 2020-11-29T16:11:10.432 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.433 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8", 2020-11-29T16:11:10.433 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.296685079Z", 2020-11-29T16:11:10.433 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.433 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.433 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.434 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.434 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.434 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.434 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:10.434 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.435 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.435 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.435 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.435 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.435 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.436 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.436 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.436 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.338804_86.1", 2020-11-29T16:11:10.436 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.436 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9", 2020-11-29T16:11:10.437 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.338804685Z", 2020-11-29T16:11:10.437 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.437 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.437 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.437 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.437 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.438 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.438 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_x_puG7-2t3lKG0zWJ2e3lY4", 2020-11-29T16:11:10.438 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.438 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.438 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.439 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.439 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.439 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.439 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.439 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.440 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.345806_87.1", 2020-11-29T16:11:10.440 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.440 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9", 2020-11-29T16:11:10.440 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.345806856Z", 2020-11-29T16:11:10.440 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.441 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.441 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.441 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.441 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.441 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.441 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_x_puG7-2t3lKG0zWJ2e3lY4", 2020-11-29T16:11:10.442 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.442 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.442 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.442 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.443 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.443 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.443 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.443 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.443 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.348268_88.1", 2020-11-29T16:11:10.444 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.444 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9", 2020-11-29T16:11:10.444 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.348268690Z", 2020-11-29T16:11:10.444 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.444 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.444 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.445 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.445 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.445 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.445 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:10.445 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.446 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.446 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.446 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.446 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.446 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.447 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.447 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.447 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.354068_89.1", 2020-11-29T16:11:10.447 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.447 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9", 2020-11-29T16:11:10.448 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.354068281Z", 2020-11-29T16:11:10.448 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.448 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.448 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.448 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.449 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.449 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.449 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:10.449 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.449 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.450 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.450 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.450 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.450 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.450 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.451 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.451 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.682590_90.1", 2020-11-29T16:11:10.451 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.451 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7", 2020-11-29T16:11:10.451 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.682590411Z", 2020-11-29T16:11:10.452 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.452 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.452 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:10.452 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.452 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.453 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.453 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:10.453 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.453 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.453 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.454 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "remove" 2020-11-29T16:11:10.454 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.454 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.454 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.454 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.455 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.689390_92.1", 2020-11-29T16:11:10.455 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.455 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7", 2020-11-29T16:11:10.455 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.689390648Z", 2020-11-29T16:11:10.455 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.455 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.456 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:10.456 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.456 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.456 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.456 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:10.457 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.457 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.457 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.460 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.460 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.461 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.461 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.461 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.461 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.764882_93.1", 2020-11-29T16:11:10.461 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.462 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8", 2020-11-29T16:11:10.462 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.764882801Z", 2020-11-29T16:11:10.462 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.462 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.462 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:10.463 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.463 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.463 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.463 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:10.463 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.464 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.464 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.464 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "remove" 2020-11-29T16:11:10.464 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.464 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.465 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.465 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.465 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.770654_94.1", 2020-11-29T16:11:10.465 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.465 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8", 2020-11-29T16:11:10.466 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.770654401Z", 2020-11-29T16:11:10.466 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.466 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.466 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:10.466 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.467 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.467 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.467 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:10.467 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.467 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.467 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.468 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.468 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.468 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.468 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.468 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.469 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.889388_95.1", 2020-11-29T16:11:10.469 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.469 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9", 2020-11-29T16:11:10.469 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.889388291Z", 2020-11-29T16:11:10.469 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.470 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.470 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:10.470 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.470 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.470 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.471 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:10.471 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.471 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.471 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.471 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "remove" 2020-11-29T16:11:10.472 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.472 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.472 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.472 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.472 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.895400_96.1", 2020-11-29T16:11:10.473 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.473 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9", 2020-11-29T16:11:10.473 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400726Z", 2020-11-29T16:11:10.473 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.473 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.474 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:10.474 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.474 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.474 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.474 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:10.475 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.475 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.475 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.475 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.475 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.476 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.476 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.476 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.476 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.191369_34.1", 2020-11-29T16:11:10.476 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.477 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10", 2020-11-29T16:11:10.477 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.191369861Z", 2020-11-29T16:11:10.477 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.477 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.477 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.478 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.478 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.478 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.478 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YrVXr7ERmibtK1P81HX79vZ", 2020-11-29T16:11:10.478 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.479 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.479 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.479 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.479 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.479 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.480 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.480 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.480 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.195873_35.1", 2020-11-29T16:11:10.480 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.480 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10", 2020-11-29T16:11:10.481 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.195873130Z", 2020-11-29T16:11:10.481 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.481 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.481 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.481 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.482 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.482 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.482 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YrVXr7ERmibtK1P81HX79vZ", 2020-11-29T16:11:10.482 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.482 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.483 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.483 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.483 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.483 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.484 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.484 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.484 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.198497_36.1", 2020-11-29T16:11:10.484 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.484 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-10", 2020-11-29T16:11:10.484 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.198497073Z", 2020-11-29T16:11:10.485 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.485 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.485 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.485 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.485 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.486 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.486 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_DFiygE422CgLw5QBztkfzu-", 2020-11-29T16:11:10.486 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.486 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.486 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.487 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.487 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.487 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.487 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.487 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.488 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.204454_37.1", 2020-11-29T16:11:10.488 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.488 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-10", 2020-11-29T16:11:10.488 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.204454835Z", 2020-11-29T16:11:10.488 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.489 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.489 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.489 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.489 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.489 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.490 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_DFiygE422CgLw5QBztkfzu-", 2020-11-29T16:11:10.490 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.490 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.490 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.490 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.491 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.491 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.491 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.491 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.491 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.229632_38.1", 2020-11-29T16:11:10.492 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.492 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11", 2020-11-29T16:11:10.492 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.229632159Z", 2020-11-29T16:11:10.492 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.492 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.493 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.493 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.493 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.493 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.493 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_20WS3iYLQJHPayxR2Iu8xxa", 2020-11-29T16:11:10.494 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.494 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.494 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.494 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.494 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.495 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.495 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.495 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.495 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.234831_39.1", 2020-11-29T16:11:10.495 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.496 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11", 2020-11-29T16:11:10.496 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.234831409Z", 2020-11-29T16:11:10.496 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.496 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.496 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.497 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.497 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.497 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.497 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_20WS3iYLQJHPayxR2Iu8xxa", 2020-11-29T16:11:10.497 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.498 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.498 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.498 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.498 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.498 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.499 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.499 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.499 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.237152_40.1", 2020-11-29T16:11:10.499 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.499 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-11", 2020-11-29T16:11:10.500 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.237152720Z", 2020-11-29T16:11:10.500 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.500 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.500 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.500 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.500 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.501 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.501 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_0w2vqVYxXcNZ-dU0Fiwn2qb", 2020-11-29T16:11:10.501 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.501 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.501 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.502 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.502 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.502 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.502 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.502 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.503 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.241905_41.1", 2020-11-29T16:11:10.503 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.503 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-11", 2020-11-29T16:11:10.503 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.241905299Z", 2020-11-29T16:11:10.503 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.504 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.504 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.504 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.504 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.504 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.505 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_0w2vqVYxXcNZ-dU0Fiwn2qb", 2020-11-29T16:11:10.505 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.505 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.505 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.505 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.506 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.506 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.506 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.506 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.506 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.311228_42.1", 2020-11-29T16:11:10.507 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.507 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12", 2020-11-29T16:11:10.507 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.311228153Z", 2020-11-29T16:11:10.507 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.507 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.508 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.508 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.508 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.508 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.508 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jrQUaYBzA_aLdZ7o_IQP6PE", 2020-11-29T16:11:10.509 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.509 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.509 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.509 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.509 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.510 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.510 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.510 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.510 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.320192_43.1", 2020-11-29T16:11:10.510 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.511 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12", 2020-11-29T16:11:10.511 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.320192323Z", 2020-11-29T16:11:10.511 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.511 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.511 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.512 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.512 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.512 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.512 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jrQUaYBzA_aLdZ7o_IQP6PE", 2020-11-29T16:11:10.512 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.513 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.513 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.513 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.513 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.513 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.514 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.514 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.514 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.328671_44.1", 2020-11-29T16:11:10.514 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.515 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-12", 2020-11-29T16:11:10.515 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.328671604Z", 2020-11-29T16:11:10.515 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.515 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.515 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.515 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.516 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.516 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.516 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5MnB5hm2ISAIj9JGdxV50H9", 2020-11-29T16:11:10.516 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.516 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.517 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.517 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.517 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.517 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.517 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.518 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.518 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.335813_45.1", 2020-11-29T16:11:10.518 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.518 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-12", 2020-11-29T16:11:10.518 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.335813349Z", 2020-11-29T16:11:10.519 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.519 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.519 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.519 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.519 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.520 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.520 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5MnB5hm2ISAIj9JGdxV50H9", 2020-11-29T16:11:10.520 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.520 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.520 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.521 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.521 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.521 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.521 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.521 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.522 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.841649_46.1", 2020-11-29T16:11:10.522 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.522 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13", 2020-11-29T16:11:10.522 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.841649100Z", 2020-11-29T16:11:10.522 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.523 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.523 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.523 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.523 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.523 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.524 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Kqyzq54nxbVSdNcuPFIuxtb", 2020-11-29T16:11:10.524 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.524 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.524 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.524 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.525 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.525 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.525 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.525 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.525 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.847258_47.1", 2020-11-29T16:11:10.526 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.526 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13", 2020-11-29T16:11:10.526 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.847258533Z", 2020-11-29T16:11:10.526 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.526 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.527 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.527 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.527 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.527 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.528 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Kqyzq54nxbVSdNcuPFIuxtb", 2020-11-29T16:11:10.528 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.528 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.528 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.528 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.529 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.529 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.529 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.529 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.529 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.849488_48.1", 2020-11-29T16:11:10.530 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.530 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-13", 2020-11-29T16:11:10.530 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.849488010Z", 2020-11-29T16:11:10.530 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.530 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.531 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.531 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.531 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.531 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.531 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rjVcqahbW1Rw3RINe760YzT", 2020-11-29T16:11:10.532 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.532 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.532 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.532 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.533 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.533 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.533 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.533 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.533 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.853552_49.1", 2020-11-29T16:11:10.534 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.534 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-13", 2020-11-29T16:11:10.534 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.853552156Z", 2020-11-29T16:11:10.534 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.534 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.535 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.535 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.535 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.535 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.536 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rjVcqahbW1Rw3RINe760YzT", 2020-11-29T16:11:10.536 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.536 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.536 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.536 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.537 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.537 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.537 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.537 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.537 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.876908_50.1", 2020-11-29T16:11:10.538 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.538 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14", 2020-11-29T16:11:10.538 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.876908996Z", 2020-11-29T16:11:10.538 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.539 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.539 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.539 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.539 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.539 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.540 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YVkRWUTasfPQtpHA7NklgCE", 2020-11-29T16:11:10.540 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.540 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.540 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.540 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.541 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.541 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.541 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.541 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.542 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.881320_51.1", 2020-11-29T16:11:10.542 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.542 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14", 2020-11-29T16:11:10.542 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.881320871Z", 2020-11-29T16:11:10.542 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.543 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.543 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.543 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.543 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.544 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.544 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YVkRWUTasfPQtpHA7NklgCE", 2020-11-29T16:11:10.544 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.544 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.544 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.545 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.545 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.545 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.545 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.545 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.546 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.883170_52.1", 2020-11-29T16:11:10.546 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.546 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-14", 2020-11-29T16:11:10.546 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.883170592Z", 2020-11-29T16:11:10.547 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.547 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.547 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.547 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.547 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.548 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.548 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AsuYPfubiijDBKRDJK6qqjt", 2020-11-29T16:11:10.548 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.548 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.548 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.549 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.549 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.549 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.549 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.550 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.550 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.887114_53.1", 2020-11-29T16:11:10.550 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.550 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-14", 2020-11-29T16:11:10.550 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.887114146Z", 2020-11-29T16:11:10.551 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.551 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.551 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.551 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.552 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.552 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.552 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AsuYPfubiijDBKRDJK6qqjt", 2020-11-29T16:11:10.552 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.552 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.553 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.553 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.553 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.553 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.553 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.554 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.554 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.932301_54.1", 2020-11-29T16:11:10.554 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.554 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15", 2020-11-29T16:11:10.554 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.932301379Z", 2020-11-29T16:11:10.555 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.555 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.555 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.555 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.556 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.556 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.556 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_x_VdKNwXX4kJXxzVcmjduF6", 2020-11-29T16:11:10.556 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.556 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.557 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.557 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.557 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.557 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.557 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.558 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.558 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.936160_55.1", 2020-11-29T16:11:10.558 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.558 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15", 2020-11-29T16:11:10.559 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.936160887Z", 2020-11-29T16:11:10.559 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.559 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.559 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.559 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.560 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.560 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.560 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_x_VdKNwXX4kJXxzVcmjduF6", 2020-11-29T16:11:10.560 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.560 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.561 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.561 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.561 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.561 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.562 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.562 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.562 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.937891_56.1", 2020-11-29T16:11:10.562 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.562 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-15", 2020-11-29T16:11:10.563 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.937891458Z", 2020-11-29T16:11:10.563 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.563 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.563 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.563 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.564 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.564 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.564 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_QoRko9LJwKUXdid0giAhASQ", 2020-11-29T16:11:10.564 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.565 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.565 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.565 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.565 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.565 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.566 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.566 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.566 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.941872_57.1", 2020-11-29T16:11:10.566 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.567 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-15", 2020-11-29T16:11:10.567 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.941872760Z", 2020-11-29T16:11:10.567 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.567 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.567 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.568 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.568 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.568 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.568 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_QoRko9LJwKUXdid0giAhASQ", 2020-11-29T16:11:10.568 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.569 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.569 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.569 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.569 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.569 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.570 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.570 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.570 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.057028_59.1", 2020-11-29T16:11:10.570 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.571 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:10.571 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.057028183Z", 2020-11-29T16:11:10.571 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.571 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.571 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.572 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.572 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.572 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.572 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:10.572 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.573 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.573 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.573 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.573 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.574 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.574 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.574 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.574 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.061373_60.1", 2020-11-29T16:11:10.574 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.575 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:10.575 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.061373004Z", 2020-11-29T16:11:10.575 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.575 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.576 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.576 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.576 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.576 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.576 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:10.577 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.577 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.577 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.577 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.577 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.578 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.578 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.578 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.578 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.063911_61.1", 2020-11-29T16:11:10.579 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.579 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16", 2020-11-29T16:11:10.579 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.063911936Z", 2020-11-29T16:11:10.579 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.579 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.580 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.580 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.580 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.580 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.580 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_qWQvPnGcRTlfP8YPDDWPU7j", 2020-11-29T16:11:10.581 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.581 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.581 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.581 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.582 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.582 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.582 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.582 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.582 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.069446_62.1", 2020-11-29T16:11:10.583 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.583 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16", 2020-11-29T16:11:10.583 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.069446267Z", 2020-11-29T16:11:10.583 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.584 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.584 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.584 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.584 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.584 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.585 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_qWQvPnGcRTlfP8YPDDWPU7j", 2020-11-29T16:11:10.585 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.585 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.585 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.585 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.586 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.586 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.586 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.586 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.097339_63.1", 2020-11-29T16:11:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.097339684Z", 2020-11-29T16:11:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.588 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.588 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.588 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.588 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.590 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.590 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.590 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.590 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.591 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.591 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.102188_64.1", 2020-11-29T16:11:10.591 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.591 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:10.591 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.102188782Z", 2020-11-29T16:11:10.592 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.592 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.592 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.592 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.593 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.593 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.593 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:10.593 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.593 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.594 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.594 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.594 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.594 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.595 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.595 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.595 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.104862_65.1", 2020-11-29T16:11:10.595 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.595 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17", 2020-11-29T16:11:10.596 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.104862905Z", 2020-11-29T16:11:10.596 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.596 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.596 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.597 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.597 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.597 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.597 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3YDKF2LZHEWfgfUO2EvjsgL", 2020-11-29T16:11:10.597 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.598 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.599 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.599 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.600 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.600 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.600 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.600 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.601 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.108626_66.1", 2020-11-29T16:11:10.601 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.601 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17", 2020-11-29T16:11:10.601 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.108626431Z", 2020-11-29T16:11:10.602 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.602 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.602 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.602 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.603 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.603 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.603 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3YDKF2LZHEWfgfUO2EvjsgL", 2020-11-29T16:11:10.603 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.603 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.604 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.604 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.604 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.604 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.604 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.605 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.605 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.150629_67.1", 2020-11-29T16:11:10.605 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.605 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:10.606 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.150629298Z", 2020-11-29T16:11:10.606 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.606 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.606 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.606 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.607 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.607 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.607 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:10.607 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.608 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.608 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.608 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.608 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.608 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.609 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.609 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.609 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.158738_68.1", 2020-11-29T16:11:10.609 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.610 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:10.610 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.158738425Z", 2020-11-29T16:11:10.610 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.610 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.610 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.611 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.611 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.611 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.611 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:10.612 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.612 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.612 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.612 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.612 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.613 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.613 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.613 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.613 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.160845_69.1", 2020-11-29T16:11:10.614 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.614 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18", 2020-11-29T16:11:10.614 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.160845058Z", 2020-11-29T16:11:10.614 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.614 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.615 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.615 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.615 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.615 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.616 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5xSDI0jeDBdQUr4kb2mMHId", 2020-11-29T16:11:10.616 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.616 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.616 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.617 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.617 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.617 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.617 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.617 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.618 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.164732_70.1", 2020-11-29T16:11:10.618 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.618 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18", 2020-11-29T16:11:10.618 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.164732376Z", 2020-11-29T16:11:10.619 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.619 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.619 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.619 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.619 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.620 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.620 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5xSDI0jeDBdQUr4kb2mMHId", 2020-11-29T16:11:10.620 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.620 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.621 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.621 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.621 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.621 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.621 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.622 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.622 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.187080_71.1", 2020-11-29T16:11:10.622 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.622 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:10.623 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.187080319Z", 2020-11-29T16:11:10.623 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.623 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.623 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:10.624 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.624 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.624 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.624 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:10.624 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.625 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.625 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.625 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.625 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.626 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.626 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.626 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.626 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.195335_72.1", 2020-11-29T16:11:10.626 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.627 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:10.627 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.195335470Z", 2020-11-29T16:11:10.627 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.627 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.627 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:10.628 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.628 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.628 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.628 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:10.629 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.629 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.629 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.629 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.630 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.630 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.630 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.630 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.630 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.203159_73.1", 2020-11-29T16:11:10.631 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.631 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:10.631 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.203159941Z", 2020-11-29T16:11:10.631 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.632 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.632 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:10.632 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.632 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.632 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.633 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:10.633 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.633 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.633 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.633 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.634 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.634 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.634 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.634 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.635 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.210320_74.1", 2020-11-29T16:11:10.635 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.635 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:10.635 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.210320129Z", 2020-11-29T16:11:10.635 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.636 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.636 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:10.636 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.636 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.636 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.637 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:10.637 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.637 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.637 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.638 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.638 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.638 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.638 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.638 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.639 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.221591_75.1", 2020-11-29T16:11:10.639 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.639 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:10.639 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.221591576Z", 2020-11-29T16:11:10.640 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.640 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.640 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:10.640 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.640 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.641 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.641 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:10.641 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.641 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.642 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.642 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.642 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.642 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.642 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.643 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.643 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.229493_76.1", 2020-11-29T16:11:10.643 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.643 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:10.644 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.229493483Z", 2020-11-29T16:11:10.644 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.644 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.644 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:10.645 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.645 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.645 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.645 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:10.645 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.646 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.646 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.646 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.646 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.646 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.647 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.647 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.647 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.967652_86.1", 2020-11-29T16:11:10.647 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.648 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:10.648 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.967652642Z", 2020-11-29T16:11:10.648 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.648 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.648 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.649 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.649 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.649 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.649 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:10.650 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.650 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.650 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.650 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.650 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.651 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.651 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.651 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.651 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.972279_87.1", 2020-11-29T16:11:10.651 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.652 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:10.652 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.972279097Z", 2020-11-29T16:11:10.652 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.652 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.653 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.653 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.653 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.653 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.653 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:10.654 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.654 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.654 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.654 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.655 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.655 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.655 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.655 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.656 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.974314_88.1", 2020-11-29T16:11:10.656 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.656 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19", 2020-11-29T16:11:10.656 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.974314737Z", 2020-11-29T16:11:10.656 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.657 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.657 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.657 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.657 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.657 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.658 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_iGyiAtOMEjJAj1PjCXsPDZz", 2020-11-29T16:11:10.658 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.658 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.658 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.659 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.659 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.659 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.659 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.659 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.660 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.978564_89.1", 2020-11-29T16:11:10.660 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.660 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19", 2020-11-29T16:11:10.660 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.978564499Z", 2020-11-29T16:11:10.661 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.661 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.661 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.661 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.661 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.662 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.662 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_iGyiAtOMEjJAj1PjCXsPDZz", 2020-11-29T16:11:10.662 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.662 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.663 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.663 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.663 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.663 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.663 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.664 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.664 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.984258_90.1", 2020-11-29T16:11:10.664 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.664 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:10.665 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.984258860Z", 2020-11-29T16:11:10.665 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.665 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.665 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:10.665 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.666 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.666 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.666 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:10.666 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.667 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.667 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.667 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.667 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.667 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.668 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.668 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.668 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.991097_91.1", 2020-11-29T16:11:10.668 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.668 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:10.669 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.991097909Z", 2020-11-29T16:11:10.669 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.669 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.669 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:10.670 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.670 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.670 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.670 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:10.671 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.671 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.671 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.671 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.671 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.672 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.672 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.672 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.672 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665997.215638_94.1", 2020-11-29T16:11:10.673 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.673 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:10.673 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.215638352Z", 2020-11-29T16:11:10.673 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.673 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.674 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.674 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.674 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.674 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.675 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.675 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.675 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.675 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.675 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.676 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.676 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.676 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.676 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.677 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665997.220413_95.1", 2020-11-29T16:11:10.677 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.677 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:10.677 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.220413426Z", 2020-11-29T16:11:10.678 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.678 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.678 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.678 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.678 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.679 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.679 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.679 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.679 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.680 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.680 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.680 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.680 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.680 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.681 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.681 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665997.222814_96.1", 2020-11-29T16:11:10.681 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.681 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20", 2020-11-29T16:11:10.682 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.222814425Z", 2020-11-29T16:11:10.682 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.682 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.682 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.682 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.683 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.683 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.683 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_caPUfpnBPovw7pnaLhfQKQu", 2020-11-29T16:11:10.683 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.684 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.684 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.684 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.684 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.684 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.685 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.685 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.685 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665997.230793_97.1", 2020-11-29T16:11:10.685 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.686 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20", 2020-11-29T16:11:10.686 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.230793956Z", 2020-11-29T16:11:10.686 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.686 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.686 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.687 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.687 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.687 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.687 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_caPUfpnBPovw7pnaLhfQKQu", 2020-11-29T16:11:10.688 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.688 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.688 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.688 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.688 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.689 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.689 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.689 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.689 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666001.819331_99.1", 2020-11-29T16:11:10.690 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.690 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:10.690 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:41.819331435Z", 2020-11-29T16:11:10.690 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.690 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.691 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.691 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.691 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.691 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.692 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.692 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.692 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.692 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.692 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.693 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.693 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.693 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.693 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.693 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666001.830749_100.1", 2020-11-29T16:11:10.694 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.694 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:10.694 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:41.830749663Z", 2020-11-29T16:11:10.694 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.695 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.695 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.695 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.695 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.695 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.696 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.696 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.696 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.696 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.696 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.697 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.697 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.697 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.697 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.698 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666061.809169_106.1", 2020-11-29T16:11:10.698 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.698 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:10.698 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:07:41.809169955Z", 2020-11-29T16:11:10.698 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.699 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.699 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.699 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.699 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.699 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.700 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.700 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:10.700 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.700 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.701 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.701 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.701 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.701 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.701 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.702 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666061.819824_107.1", 2020-11-29T16:11:10.702 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.702 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:10.702 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:07:41.819824059Z", 2020-11-29T16:11:10.703 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.703 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.703 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.703 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.703 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.704 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.704 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:10.704 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:10.704 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.705 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.705 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.705 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.705 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.705 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.706 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.706 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.732730_108.1", 2020-11-29T16:11:10.706 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.706 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:10.706 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.732730870Z", 2020-11-29T16:11:10.707 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.707 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.707 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.707 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.707 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.708 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.708 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:10.708 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.708 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.708 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.709 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.709 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.709 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.709 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.710 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.710 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.738936_109.1", 2020-11-29T16:11:10.710 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.710 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:10.710 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.738936839Z", 2020-11-29T16:11:10.711 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.711 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.711 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.711 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.712 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.712 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.712 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:10.712 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.713 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.713 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.713 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.713 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.713 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.714 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.714 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.714 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.740862_110.1", 2020-11-29T16:11:10.714 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.715 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21", 2020-11-29T16:11:10.715 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.740862527Z", 2020-11-29T16:11:10.715 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.715 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.715 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.716 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.716 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.716 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.716 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1LtU-sUHWZQfFCaHHia4XL6", 2020-11-29T16:11:10.717 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.717 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.717 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.717 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.717 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.718 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.718 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.718 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.718 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.745932_111.1", 2020-11-29T16:11:10.718 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.719 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21", 2020-11-29T16:11:10.719 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.745932979Z", 2020-11-29T16:11:10.719 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.719 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.720 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.720 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.720 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.720 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.720 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1LtU-sUHWZQfFCaHHia4XL6", 2020-11-29T16:11:10.721 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.721 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.721 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.721 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.722 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.722 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.722 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.722 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.722 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.771777_112.1", 2020-11-29T16:11:10.723 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.723 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:10.723 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.771777245Z", 2020-11-29T16:11:10.723 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.724 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.724 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.724 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.724 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.724 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.725 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:10.725 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.725 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.725 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.726 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.726 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.726 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.726 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.726 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.727 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.779214_113.1", 2020-11-29T16:11:10.727 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.727 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:10.727 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.779214800Z", 2020-11-29T16:11:10.728 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.728 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.728 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.728 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.729 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.729 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.729 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:10.729 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.729 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.730 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.730 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.730 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.730 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.731 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.731 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.731 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.781033_114.1", 2020-11-29T16:11:10.731 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.732 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22", 2020-11-29T16:11:10.732 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.781033305Z", 2020-11-29T16:11:10.732 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.732 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.732 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.733 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.733 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.733 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.733 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_yAIfpIe3NidR6qhQ52Y6gQ9", 2020-11-29T16:11:10.734 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.734 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.734 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.734 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.734 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.735 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.735 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.735 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.735 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.786435_115.1", 2020-11-29T16:11:10.736 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.736 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22", 2020-11-29T16:11:10.736 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.786435846Z", 2020-11-29T16:11:10.736 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.737 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.737 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.737 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.737 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.737 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.738 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_yAIfpIe3NidR6qhQ52Y6gQ9", 2020-11-29T16:11:10.738 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.738 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.738 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.738 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.739 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.739 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.739 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.739 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.740 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.833583_116.1", 2020-11-29T16:11:10.740 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.740 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:10.740 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.833583970Z", 2020-11-29T16:11:10.741 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.741 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.741 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.741 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.741 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.742 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.742 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:10.742 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.742 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.743 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.743 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.743 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.743 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.743 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.744 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.744 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.838249_117.1", 2020-11-29T16:11:10.744 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.744 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:10.745 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.838249602Z", 2020-11-29T16:11:10.745 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.745 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.745 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.745 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.746 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.746 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.746 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:10.746 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.747 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.747 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.747 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.747 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.747 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.748 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.748 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.748 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.840897_118.1", 2020-11-29T16:11:10.748 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.748 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23", 2020-11-29T16:11:10.749 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.840897485Z", 2020-11-29T16:11:10.749 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.749 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.749 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.750 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.750 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.750 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.750 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_yUDBs52ibfejiizLQQANf-x", 2020-11-29T16:11:10.751 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.751 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.751 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.751 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.751 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.752 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.752 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.752 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.752 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.846339_119.1", 2020-11-29T16:11:10.753 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.753 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23", 2020-11-29T16:11:10.753 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.846339269Z", 2020-11-29T16:11:10.753 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.754 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.754 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.754 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.754 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.754 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.755 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_yUDBs52ibfejiizLQQANf-x", 2020-11-29T16:11:10.755 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.755 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.755 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.756 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.756 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.756 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.756 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.756 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.757 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.869230_120.1", 2020-11-29T16:11:10.757 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.757 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:10.757 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.869230022Z", 2020-11-29T16:11:10.757 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.758 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.758 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:10.758 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.758 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.759 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.759 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:10.759 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.759 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.759 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.760 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.760 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.760 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.760 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.760 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.761 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.875173_121.1", 2020-11-29T16:11:10.761 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.761 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:10.761 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.875173699Z", 2020-11-29T16:11:10.761 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.762 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.762 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:10.762 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.762 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.762 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.763 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:10.763 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.763 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.763 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.763 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.764 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.764 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.764 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.764 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.764 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.879784_122.1", 2020-11-29T16:11:10.765 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.765 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:10.765 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.879784537Z", 2020-11-29T16:11:10.765 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.765 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.766 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:10.766 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.766 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.766 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.767 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:10.767 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.767 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.767 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.767 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.768 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.768 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.768 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.768 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.768 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.886953_123.1", 2020-11-29T16:11:10.769 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.769 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:10.769 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.886953932Z", 2020-11-29T16:11:10.769 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.769 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.770 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:10.770 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.770 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.770 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.770 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:10.771 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.771 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.771 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.771 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.771 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.772 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.772 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.772 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.772 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.901158_124.1", 2020-11-29T16:11:10.772 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.773 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:10.773 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.901158656Z", 2020-11-29T16:11:10.773 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.773 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.773 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:10.774 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.774 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.774 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.774 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:10.775 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.775 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.775 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.775 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.775 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.776 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.776 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.776 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.776 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.915755_125.1", 2020-11-29T16:11:10.776 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.777 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:10.777 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.915755723Z", 2020-11-29T16:11:10.777 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.777 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.777 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:10.778 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.778 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.778 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.778 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:10.778 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.779 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.779 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.779 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.779 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.779 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.780 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.780 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.780 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.456443_126.1", 2020-11-29T16:11:10.780 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.780 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:10.781 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.456443611Z", 2020-11-29T16:11:10.781 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.781 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.781 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.781 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.782 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.782 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.782 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.782 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.783 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.783 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.783 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.783 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.783 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.784 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.784 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.784 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.460387_127.1", 2020-11-29T16:11:10.784 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.784 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:10.785 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.460387996Z", 2020-11-29T16:11:10.785 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.785 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.785 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.785 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.786 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.786 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.786 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.786 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.786 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.787 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.787 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.787 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.787 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.787 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.788 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.788 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.462193_128.1", 2020-11-29T16:11:10.788 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.788 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24", 2020-11-29T16:11:10.788 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.462193185Z", 2020-11-29T16:11:10.789 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.789 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.789 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.789 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.789 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.790 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.790 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_8rFe9Ex-HuV_dw_B2ecHBhB", 2020-11-29T16:11:10.790 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.790 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.791 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.791 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.791 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.791 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.791 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.792 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.792 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.467714_129.1", 2020-11-29T16:11:10.792 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.792 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24", 2020-11-29T16:11:10.792 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.467714061Z", 2020-11-29T16:11:10.793 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.793 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.793 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.793 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.793 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.794 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.794 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_8rFe9Ex-HuV_dw_B2ecHBhB", 2020-11-29T16:11:10.794 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.794 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.794 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.795 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.795 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.795 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.795 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.795 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.796 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.490669_130.1", 2020-11-29T16:11:10.796 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.796 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:10.796 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.490669098Z", 2020-11-29T16:11:10.796 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.797 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.797 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.797 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.797 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.797 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.798 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.798 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.798 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.798 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.798 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.799 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.799 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.799 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.799 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.799 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.496194_131.1", 2020-11-29T16:11:10.800 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.800 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:10.800 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.496194848Z", 2020-11-29T16:11:10.800 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.800 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.801 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.801 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.801 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.801 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.801 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.802 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.802 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.802 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.802 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.803 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.803 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.803 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.803 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.803 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.498339_132.1", 2020-11-29T16:11:10.804 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.804 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25", 2020-11-29T16:11:10.804 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.498339597Z", 2020-11-29T16:11:10.804 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.804 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.805 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.805 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.805 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.805 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.805 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_s63DKGRwg1GnITE9VweheW4", 2020-11-29T16:11:10.806 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.806 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.806 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.806 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.806 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.807 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.807 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.807 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.807 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.502532_133.1", 2020-11-29T16:11:10.807 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.808 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25", 2020-11-29T16:11:10.808 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.502532461Z", 2020-11-29T16:11:10.808 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.808 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.808 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.809 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.809 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.809 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.809 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_s63DKGRwg1GnITE9VweheW4", 2020-11-29T16:11:10.809 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.810 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.810 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.810 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.810 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.810 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.811 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.811 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.811 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.557238_134.1", 2020-11-29T16:11:10.811 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.811 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:10.812 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.557238717Z", 2020-11-29T16:11:10.812 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.812 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.812 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.812 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.813 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.813 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.813 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.813 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.814 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.814 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.814 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.814 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.815 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.815 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.815 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.815 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.562930_135.1", 2020-11-29T16:11:10.816 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.816 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:10.816 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.562930060Z", 2020-11-29T16:11:10.816 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.817 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.817 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.817 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.817 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.818 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.818 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.818 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.819 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.819 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.819 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.819 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.820 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.820 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.820 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.820 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.565930_136.1", 2020-11-29T16:11:10.820 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.821 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26", 2020-11-29T16:11:10.821 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.565930147Z", 2020-11-29T16:11:10.821 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.821 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.821 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.822 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.822 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.822 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.822 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_HEKlnFCl4gKqxl9FXC6_Slr", 2020-11-29T16:11:10.823 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.823 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.823 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.823 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.823 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.824 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.824 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.824 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.824 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.574718_137.1", 2020-11-29T16:11:10.824 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.825 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26", 2020-11-29T16:11:10.825 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.574718238Z", 2020-11-29T16:11:10.825 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.825 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.825 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.826 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.826 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.826 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.826 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_HEKlnFCl4gKqxl9FXC6_Slr", 2020-11-29T16:11:10.826 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.827 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.827 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.827 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.827 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.827 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.828 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.828 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.828 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.594772_138.1", 2020-11-29T16:11:10.828 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.828 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:10.829 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.594772332Z", 2020-11-29T16:11:10.829 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.829 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.829 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.829 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.830 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.830 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.830 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.830 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.830 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.831 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.831 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.831 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.831 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.831 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.832 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.832 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.599622_139.1", 2020-11-29T16:11:10.832 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.832 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:10.832 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.599622375Z", 2020-11-29T16:11:10.833 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.833 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.833 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.833 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.833 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.834 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.834 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.834 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.834 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.834 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.835 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.835 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.835 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.835 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.835 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.836 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.603089_140.1", 2020-11-29T16:11:10.836 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.836 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:10.836 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.603089353Z", 2020-11-29T16:11:10.836 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.837 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.837 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.837 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.837 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.837 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.838 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.838 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:10.838 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.838 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.838 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.839 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.839 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.839 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.839 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.839 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.612024_141.1", 2020-11-29T16:11:10.840 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.840 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:10.840 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.612024636Z", 2020-11-29T16:11:10.840 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.841 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.841 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.841 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.841 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.841 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.842 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:10.842 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:10.842 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.842 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.842 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.843 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.843 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.843 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.843 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.843 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.616582_142.1", 2020-11-29T16:11:10.844 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.844 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:10.844 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.616582099Z", 2020-11-29T16:11:10.844 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.844 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.845 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.845 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.845 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.845 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.845 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.846 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.846 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.846 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.846 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.846 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.847 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.847 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.847 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.847 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.624065_143.1", 2020-11-29T16:11:10.847 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.848 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:10.848 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.624065742Z", 2020-11-29T16:11:10.848 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.848 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.849 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.849 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.849 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.849 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.849 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.850 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.850 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.850 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.850 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.850 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.851 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.851 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.851 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.851 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.633769_144.1", 2020-11-29T16:11:10.851 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.852 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:10.852 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.633769558Z", 2020-11-29T16:11:10.852 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.852 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.852 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.853 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.853 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.853 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.853 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.853 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:10.854 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.854 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.854 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.854 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.854 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.855 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.855 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.855 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.641875_145.1", 2020-11-29T16:11:10.855 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.855 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:10.856 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.641875589Z", 2020-11-29T16:11:10.856 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.856 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.856 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.856 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.857 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.857 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.857 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:10.857 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:10.857 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.858 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.858 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.858 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.858 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.859 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.859 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.859 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.655938_146.1", 2020-11-29T16:11:10.859 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.859 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:10.860 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.655938546Z", 2020-11-29T16:11:10.860 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.860 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.860 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.860 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.861 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.861 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.861 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.861 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.861 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.862 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.862 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.862 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.862 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.862 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.863 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.863 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.665066_147.1", 2020-11-29T16:11:10.863 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.863 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:10.863 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.665066109Z", 2020-11-29T16:11:10.864 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.864 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.864 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.864 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.864 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.865 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.865 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.865 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.865 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.865 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.866 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.866 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.866 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.866 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.866 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.867 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.677379_148.1", 2020-11-29T16:11:10.867 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.867 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:10.867 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.677379068Z", 2020-11-29T16:11:10.867 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.868 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.868 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.868 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.868 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.868 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.869 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.869 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:10.869 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.869 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.870 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.870 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.870 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.870 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.870 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.871 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.684234_149.1", 2020-11-29T16:11:10.871 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.871 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:10.871 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.684234315Z", 2020-11-29T16:11:10.871 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.872 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.872 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.872 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.872 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.872 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.873 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:10.873 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:10.873 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.873 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.873 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.874 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.874 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.874 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.874 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.874 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.338993_154.1", 2020-11-29T16:11:10.875 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.875 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:10.875 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.338993057Z", 2020-11-29T16:11:10.875 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.875 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.876 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.876 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.876 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.876 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.876 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:10.877 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.877 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.877 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.877 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.877 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.878 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.878 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.878 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.878 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.346126_155.1", 2020-11-29T16:11:10.879 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.879 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:10.879 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.346126383Z", 2020-11-29T16:11:10.879 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.879 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.880 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.880 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.880 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.880 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.880 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:10.881 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.881 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.881 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.881 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.881 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.882 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.882 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.882 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.882 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.348222_156.1", 2020-11-29T16:11:10.882 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.883 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27", 2020-11-29T16:11:10.883 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.348222396Z", 2020-11-29T16:11:10.883 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.883 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.883 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.884 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.884 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.884 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.884 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Wc4auwc30IepX657G2eZM-u", 2020-11-29T16:11:10.885 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.885 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.885 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.885 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.885 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.886 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.886 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.886 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.886 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.353951_157.1", 2020-11-29T16:11:10.886 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.887 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27", 2020-11-29T16:11:10.887 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.353951918Z", 2020-11-29T16:11:10.887 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.887 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.887 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.888 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.888 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.888 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.888 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Wc4auwc30IepX657G2eZM-u", 2020-11-29T16:11:10.888 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.889 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.889 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.889 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.889 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.890 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.890 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.890 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.890 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.390106_158.1", 2020-11-29T16:11:10.890 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.891 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:10.891 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.390106865Z", 2020-11-29T16:11:10.891 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.891 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.891 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.892 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.892 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.892 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.892 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:10.892 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.893 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.893 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.893 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.893 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.893 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.894 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.894 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.894 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.397881_159.1", 2020-11-29T16:11:10.894 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.895 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:10.895 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.397881409Z", 2020-11-29T16:11:10.895 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.895 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.895 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.896 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.896 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.896 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.896 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:10.896 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.897 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.897 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.897 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.897 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.897 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.898 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.898 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.898 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.400119_160.1", 2020-11-29T16:11:10.898 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.899 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28", 2020-11-29T16:11:10.899 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.400119169Z", 2020-11-29T16:11:10.899 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.899 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.899 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.900 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.900 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.900 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.900 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vorB3p52YuzxIbSF-GHscIz", 2020-11-29T16:11:10.900 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.901 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.901 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.901 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.901 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.901 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.902 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.902 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.902 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.404155_161.1", 2020-11-29T16:11:10.902 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.902 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28", 2020-11-29T16:11:10.903 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.404155690Z", 2020-11-29T16:11:10.903 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.903 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.903 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.903 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.904 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.904 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.904 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vorB3p52YuzxIbSF-GHscIz", 2020-11-29T16:11:10.904 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.905 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.905 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.905 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.905 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.905 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.906 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.906 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.906 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.461726_162.1", 2020-11-29T16:11:10.906 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.906 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:10.907 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.461726137Z", 2020-11-29T16:11:10.907 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.907 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.907 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.907 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.908 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.908 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.908 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:10.908 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.908 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.909 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.909 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.909 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.909 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.909 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.910 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.910 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.466239_163.1", 2020-11-29T16:11:10.910 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.910 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:10.911 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.466239727Z", 2020-11-29T16:11:10.911 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.911 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.911 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.911 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.912 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.912 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.912 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:10.912 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.912 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.913 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.913 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.913 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.913 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.913 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.914 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.914 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.471274_164.1", 2020-11-29T16:11:10.914 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.914 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29", 2020-11-29T16:11:10.914 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.471274792Z", 2020-11-29T16:11:10.915 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.915 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.915 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.915 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.916 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.916 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.916 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tW9vOLbEcF0UPbuJYTP9PYC", 2020-11-29T16:11:10.916 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.916 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.917 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.917 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.917 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.917 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.917 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.918 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.918 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.475990_165.1", 2020-11-29T16:11:10.918 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.918 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29", 2020-11-29T16:11:10.918 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.475990005Z", 2020-11-29T16:11:10.919 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.919 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.919 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.919 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.919 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.920 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.920 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tW9vOLbEcF0UPbuJYTP9PYC", 2020-11-29T16:11:10.920 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.920 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.921 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.921 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.921 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.921 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.921 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.922 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.922 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.500752_166.1", 2020-11-29T16:11:10.922 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.922 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:10.922 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.500752451Z", 2020-11-29T16:11:10.923 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.923 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.923 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:10.923 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.923 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.924 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.924 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:10.924 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.924 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.924 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.925 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.925 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.925 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.925 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.925 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.926 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.506749_167.1", 2020-11-29T16:11:10.926 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.926 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:10.926 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.506749250Z", 2020-11-29T16:11:10.927 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.927 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.927 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:10.927 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.927 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.928 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.928 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:10.928 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.928 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.928 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.929 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.929 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.929 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.929 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.929 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.930 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.561039_168.1", 2020-11-29T16:11:10.930 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.930 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:10.930 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.561039789Z", 2020-11-29T16:11:10.930 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.931 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.931 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:10.931 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.931 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.931 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.932 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:10.932 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.932 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.932 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.932 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.933 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.933 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.933 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.933 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.933 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.568104_169.1", 2020-11-29T16:11:10.934 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.934 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:10.934 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.568104840Z", 2020-11-29T16:11:10.934 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.935 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.935 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:10.935 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.935 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.935 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.936 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:10.936 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.936 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.936 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.936 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.937 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.937 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.937 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.937 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.937 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.624595_170.1", 2020-11-29T16:11:10.938 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.938 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:10.938 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.624595921Z", 2020-11-29T16:11:10.938 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.938 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.939 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:10.939 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.939 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.939 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.939 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:10.940 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.940 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.940 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.940 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.940 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.941 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.941 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.941 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.941 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.631524_171.1", 2020-11-29T16:11:10.941 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.942 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:10.942 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.631524395Z", 2020-11-29T16:11:10.942 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.942 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.942 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:10.943 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.943 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.943 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.943 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:10.943 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.944 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.944 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.944 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.944 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.944 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.945 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.945 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.945 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.709256_172.1", 2020-11-29T16:11:10.945 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.946 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:10.946 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.709256940Z", 2020-11-29T16:11:10.946 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.946 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.946 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.947 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.947 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.947 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.947 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:10.947 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.948 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.948 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.948 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.948 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.948 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.949 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.949 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.949 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.713773_173.1", 2020-11-29T16:11:10.949 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.949 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:10.950 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.713773728Z", 2020-11-29T16:11:10.950 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.950 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.950 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.950 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.951 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.951 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.951 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:10.951 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.952 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.952 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.952 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.952 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.952 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.953 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.953 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.953 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.718379_174.1", 2020-11-29T16:11:10.953 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.953 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30", 2020-11-29T16:11:10.954 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.718379824Z", 2020-11-29T16:11:10.954 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.954 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.954 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.954 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.955 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.955 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.955 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_h_WBzhTZe664PHorKv6_Am0", 2020-11-29T16:11:10.955 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.955 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.956 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.956 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.956 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.956 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.956 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.957 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.957 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.723686_175.1", 2020-11-29T16:11:10.957 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.957 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30", 2020-11-29T16:11:10.957 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.723686356Z", 2020-11-29T16:11:10.958 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.958 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.958 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.958 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.958 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.959 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.959 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_h_WBzhTZe664PHorKv6_Am0", 2020-11-29T16:11:10.959 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.959 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.960 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.960 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.960 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.960 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.960 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.961 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.961 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.752571_176.1", 2020-11-29T16:11:10.961 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.961 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:10.961 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.752571930Z", 2020-11-29T16:11:10.962 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.962 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.962 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.962 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.962 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.963 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.963 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:10.963 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.963 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.963 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.964 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.964 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.964 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.964 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.964 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.965 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.756904_177.1", 2020-11-29T16:11:10.965 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.965 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:10.965 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.756904681Z", 2020-11-29T16:11:10.965 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.966 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.966 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.966 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.966 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.966 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.967 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:10.967 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.967 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.967 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.968 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.968 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.968 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.968 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.968 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.969 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.759821_178.1", 2020-11-29T16:11:10.969 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.969 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31", 2020-11-29T16:11:10.969 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.759821922Z", 2020-11-29T16:11:10.969 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.970 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.970 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.970 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.970 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.970 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.971 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_eKjXcKGmg5yyLHdihUsR2sW", 2020-11-29T16:11:10.971 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.971 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.971 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.971 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.972 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.972 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.972 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.972 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.972 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.764291_179.1", 2020-11-29T16:11:10.973 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.973 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31", 2020-11-29T16:11:10.973 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.764291393Z", 2020-11-29T16:11:10.973 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.973 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.974 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.974 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.974 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.974 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.974 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_eKjXcKGmg5yyLHdihUsR2sW", 2020-11-29T16:11:10.975 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.975 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.975 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.975 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.976 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.976 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.976 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.976 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.976 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.825137_180.1", 2020-11-29T16:11:10.977 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.977 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:10.977 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.825137534Z", 2020-11-29T16:11:10.977 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.977 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.978 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.978 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.978 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.978 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.978 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:10.979 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.979 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.979 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.979 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.979 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.980 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.980 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.980 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.980 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.829684_181.1", 2020-11-29T16:11:10.980 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.981 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:10.981 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.829684939Z", 2020-11-29T16:11:10.981 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.981 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.981 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.982 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.982 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.982 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.982 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:10.982 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.983 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.983 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.983 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.983 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.983 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.984 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.984 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.984 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.831781_182.1", 2020-11-29T16:11:10.984 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.984 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32", 2020-11-29T16:11:10.985 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.831781072Z", 2020-11-29T16:11:10.985 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.985 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.985 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.985 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.986 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.986 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.986 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_dhvj1N7nCxk5qukpVfy3IjW", 2020-11-29T16:11:10.986 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.987 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.987 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.987 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.987 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.987 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.988 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.988 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.988 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.835966_183.1", 2020-11-29T16:11:10.988 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:10.988 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32", 2020-11-29T16:11:10.989 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.835966939Z", 2020-11-29T16:11:10.989 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.989 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.989 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:10.989 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:10.990 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.990 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.990 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_dhvj1N7nCxk5qukpVfy3IjW", 2020-11-29T16:11:10.990 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.990 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.991 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.991 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.991 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.991 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.991 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.992 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.992 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.860087_184.1", 2020-11-29T16:11:10.992 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.992 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:10.992 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.860087065Z", 2020-11-29T16:11:10.993 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.993 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.993 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:10.993 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.993 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.994 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.994 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:10.994 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.994 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.994 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.995 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:10.995 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.995 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.995 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.995 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:10.996 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.865870_185.1", 2020-11-29T16:11:10.996 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:10.996 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:10.996 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.865870483Z", 2020-11-29T16:11:10.996 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:10.997 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:10.997 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:10.997 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:10.997 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.997 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:10.998 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:10.998 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:10.998 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.998 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:10.999 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:10.999 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.999 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:10.999 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:10.999 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.000 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.914709_186.1", 2020-11-29T16:11:11.000 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.000 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:11.000 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.914709667Z", 2020-11-29T16:11:11.000 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.001 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.001 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:11.001 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.001 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.001 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.002 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:11.002 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.002 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.002 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.002 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.003 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.003 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.003 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.003 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.003 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.925154_187.1", 2020-11-29T16:11:11.004 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.004 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:11.004 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.925154339Z", 2020-11-29T16:11:11.004 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.004 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.005 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:11.005 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.005 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.005 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.005 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:11.006 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.006 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.006 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.006 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.007 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.007 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.007 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.007 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.007 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.991321_188.1", 2020-11-29T16:11:11.008 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.008 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:11.008 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.991321647Z", 2020-11-29T16:11:11.008 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.008 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.009 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:11.009 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.009 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.009 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.009 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:11.010 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.010 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.010 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.010 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.010 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.011 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.011 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.011 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.011 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.999983_189.1", 2020-11-29T16:11:11.012 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.012 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:11.012 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.999983004Z", 2020-11-29T16:11:11.012 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.012 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.013 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:11.013 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.013 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.013 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.013 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:11.014 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.014 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.014 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.014 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.014 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.015 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.015 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.015 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.015 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.076605_190.1", 2020-11-29T16:11:11.015 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.016 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:11.016 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.076605830Z", 2020-11-29T16:11:11.016 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.016 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.016 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.017 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.017 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.017 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.017 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:11.018 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.018 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.018 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.018 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.018 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.019 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.019 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.019 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.019 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.081952_191.1", 2020-11-29T16:11:11.019 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.020 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:11.020 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.081952008Z", 2020-11-29T16:11:11.020 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.020 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.020 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.021 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.021 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.021 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.021 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:11.021 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.022 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.022 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.022 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.022 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.022 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.023 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.023 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.023 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.083820_192.1", 2020-11-29T16:11:11.023 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.024 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33", 2020-11-29T16:11:11.024 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.083820333Z", 2020-11-29T16:11:11.024 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.024 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.024 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.025 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.025 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.025 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.025 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_aScNky5UrTuH12xWR79F83e", 2020-11-29T16:11:11.025 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.026 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.026 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.026 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.026 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.026 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.027 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.027 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.027 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.088982_193.1", 2020-11-29T16:11:11.027 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.027 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33", 2020-11-29T16:11:11.028 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.088982559Z", 2020-11-29T16:11:11.028 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.028 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.028 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.028 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.029 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.029 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.029 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_aScNky5UrTuH12xWR79F83e", 2020-11-29T16:11:11.029 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.030 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.030 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.030 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.030 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.030 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.031 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.031 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.031 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.121086_194.1", 2020-11-29T16:11:11.031 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.031 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:11.032 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.121086133Z", 2020-11-29T16:11:11.032 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.032 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.032 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.032 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.033 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.033 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.033 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:11.033 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.033 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.034 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.034 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.034 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.034 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.035 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.035 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.035 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.133205_195.1", 2020-11-29T16:11:11.035 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.035 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:11.036 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.133205341Z", 2020-11-29T16:11:11.036 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.036 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.036 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.036 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.037 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.037 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.037 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:11.037 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.037 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.038 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.038 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.038 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.038 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.039 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.039 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.039 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.138006_196.1", 2020-11-29T16:11:11.039 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.039 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34", 2020-11-29T16:11:11.040 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.138006291Z", 2020-11-29T16:11:11.040 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.040 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.040 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.040 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.041 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.041 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.041 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_9r_K5jlLAncl40ETfoVeJjK", 2020-11-29T16:11:11.041 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.041 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.042 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.042 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.042 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.042 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.042 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.043 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.043 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.153529_197.1", 2020-11-29T16:11:11.043 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.043 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34", 2020-11-29T16:11:11.044 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.153529708Z", 2020-11-29T16:11:11.044 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.044 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.044 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.044 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.045 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.045 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.045 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_9r_K5jlLAncl40ETfoVeJjK", 2020-11-29T16:11:11.045 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.045 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.046 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.046 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.046 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.046 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.046 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.047 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.047 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.225476_198.1", 2020-11-29T16:11:11.047 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.047 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:11.047 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.225476965Z", 2020-11-29T16:11:11.048 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.048 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.048 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.048 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.049 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.049 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.049 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:11.049 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.049 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.050 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.050 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.050 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.050 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.050 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.051 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.051 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.230332_199.1", 2020-11-29T16:11:11.051 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.051 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:11.051 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.230332153Z", 2020-11-29T16:11:11.052 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.052 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.052 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.052 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.052 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.053 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.053 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:11.053 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.053 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.054 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.054 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.054 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.054 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.054 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.055 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.055 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.232895_200.1", 2020-11-29T16:11:11.055 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.055 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35", 2020-11-29T16:11:11.055 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.232895662Z", 2020-11-29T16:11:11.056 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.056 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.056 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.056 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.056 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.057 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.057 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_BauiNjhDLe4JD_SZUYzW5BH", 2020-11-29T16:11:11.057 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.057 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.057 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.058 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.058 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.058 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.058 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.058 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.059 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.237930_201.1", 2020-11-29T16:11:11.059 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.059 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35", 2020-11-29T16:11:11.059 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.237930233Z", 2020-11-29T16:11:11.059 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.060 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.060 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.060 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.060 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.060 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.061 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_BauiNjhDLe4JD_SZUYzW5BH", 2020-11-29T16:11:11.061 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.061 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.061 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.061 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.062 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.062 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.062 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.062 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.063 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.261144_202.1", 2020-11-29T16:11:11.063 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.063 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:11.063 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.261144444Z", 2020-11-29T16:11:11.063 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.064 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.064 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:11.064 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.064 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.064 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.065 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:11.065 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.065 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.065 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.065 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.066 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.066 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.066 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.066 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.066 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.266462_203.1", 2020-11-29T16:11:11.067 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.067 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:11.067 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.266462524Z", 2020-11-29T16:11:11.067 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.067 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.068 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:11.068 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.068 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.068 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.068 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:11.069 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.069 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.069 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.069 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.069 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.070 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.070 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.070 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.070 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666124.313715_204.1", 2020-11-29T16:11:11.070 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.071 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:11.071 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:44.313715914Z", 2020-11-29T16:11:11.071 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.071 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.071 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:11.072 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.072 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.072 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.072 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:11.072 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.073 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.073 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.073 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.073 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.074 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.074 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.074 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.074 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666124.327834_205.1", 2020-11-29T16:11:11.074 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.075 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:11.075 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:44.327834836Z", 2020-11-29T16:11:11.075 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.075 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.075 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:11.076 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.076 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.076 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.076 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:11.076 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.077 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.077 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.077 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.077 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.077 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.078 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.078 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.078 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.389734_206.1", 2020-11-29T16:11:11.078 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.079 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:11.079 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.389734131Z", 2020-11-29T16:11:11.079 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.079 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.079 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:11.080 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.080 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.080 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.080 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:11.080 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.081 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.081 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.081 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.081 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.081 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.082 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.082 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.082 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.399959_207.1", 2020-11-29T16:11:11.082 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.082 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:11.083 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.399959602Z", 2020-11-29T16:11:11.083 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.083 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.083 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:11.083 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.084 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.084 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.084 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:11.084 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:11.085 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.085 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.085 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.085 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.086 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.086 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.086 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.086 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.480704_208.1", 2020-11-29T16:11:11.087 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.087 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36", 2020-11-29T16:11:11.087 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.480704401Z", 2020-11-29T16:11:11.087 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.088 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.088 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.088 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.088 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.088 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.089 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_UJxOrACWaVFo5tZbkaJd5aB", 2020-11-29T16:11:11.089 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.089 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.089 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.090 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.090 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.090 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.090 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.091 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.091 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.485309_209.1", 2020-11-29T16:11:11.091 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.091 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36", 2020-11-29T16:11:11.092 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.485309628Z", 2020-11-29T16:11:11.092 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.092 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.092 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.092 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.093 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.093 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.093 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_UJxOrACWaVFo5tZbkaJd5aB", 2020-11-29T16:11:11.093 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.093 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.094 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.094 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.094 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.094 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.095 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.095 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.095 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.487975_210.1", 2020-11-29T16:11:11.095 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.095 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-36", 2020-11-29T16:11:11.096 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.487975960Z", 2020-11-29T16:11:11.096 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.096 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.096 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.096 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.097 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.097 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.097 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VZJxPfFqHBrvBon1FcGTxPs", 2020-11-29T16:11:11.097 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.097 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.098 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.098 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.098 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.098 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.098 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.099 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.099 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.491713_211.1", 2020-11-29T16:11:11.099 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.099 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-36", 2020-11-29T16:11:11.099 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.491713665Z", 2020-11-29T16:11:11.100 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.100 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.100 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.100 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.101 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.101 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.101 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VZJxPfFqHBrvBon1FcGTxPs", 2020-11-29T16:11:11.101 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.101 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.102 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.102 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.102 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.102 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.102 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.103 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.103 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.525177_212.1", 2020-11-29T16:11:11.103 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.103 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37", 2020-11-29T16:11:11.103 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.525177090Z", 2020-11-29T16:11:11.104 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.104 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.104 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.104 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.104 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.105 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.105 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_syk3UA8fSzEvnh7yGHxS7iU", 2020-11-29T16:11:11.105 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.105 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.105 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.106 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.106 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.106 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.106 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.106 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.107 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.534079_213.1", 2020-11-29T16:11:11.107 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.107 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37", 2020-11-29T16:11:11.107 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.534079771Z", 2020-11-29T16:11:11.108 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.108 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.108 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.108 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.108 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.109 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.109 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_syk3UA8fSzEvnh7yGHxS7iU", 2020-11-29T16:11:11.109 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.109 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.109 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.110 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.110 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.110 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.110 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.110 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.111 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.540941_214.1", 2020-11-29T16:11:11.111 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.111 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-37", 2020-11-29T16:11:11.111 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.540941786Z", 2020-11-29T16:11:11.111 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.112 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.112 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.112 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.112 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.113 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.113 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_LPLM1eQYX53fxff8emhXqjO", 2020-11-29T16:11:11.113 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.113 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.113 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.114 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.114 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.114 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.114 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.114 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.115 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.547825_215.1", 2020-11-29T16:11:11.115 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.115 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-37", 2020-11-29T16:11:11.115 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.547825214Z", 2020-11-29T16:11:11.115 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.116 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.116 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.116 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.116 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.116 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.117 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_LPLM1eQYX53fxff8emhXqjO", 2020-11-29T16:11:11.117 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.117 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.117 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.117 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.118 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.118 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.118 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.118 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.118 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.590883_216.1", 2020-11-29T16:11:11.119 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.119 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38", 2020-11-29T16:11:11.119 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.590883464Z", 2020-11-29T16:11:11.119 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.119 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.120 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.120 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.120 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.120 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.120 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_sluiDwZnW0jlavok--xXUQM", 2020-11-29T16:11:11.121 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.121 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.121 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.121 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.121 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.122 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.122 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.122 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.122 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.595783_217.1", 2020-11-29T16:11:11.123 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.123 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38", 2020-11-29T16:11:11.123 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.595783563Z", 2020-11-29T16:11:11.123 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.123 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.124 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.124 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.124 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.124 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.124 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_sluiDwZnW0jlavok--xXUQM", 2020-11-29T16:11:11.125 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.125 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.125 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.125 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.125 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.126 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.126 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.126 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.126 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.597926_218.1", 2020-11-29T16:11:11.126 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.127 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-38", 2020-11-29T16:11:11.127 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.597926148Z", 2020-11-29T16:11:11.127 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.127 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.127 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.128 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.128 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.128 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.128 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_s5CCY0lZsRXaG97sebP-L5L", 2020-11-29T16:11:11.128 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.129 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.129 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.129 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.129 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.129 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.130 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.130 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.130 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.602582_219.1", 2020-11-29T16:11:11.130 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.131 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-38", 2020-11-29T16:11:11.131 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.602582602Z", 2020-11-29T16:11:11.131 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.131 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.131 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.132 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.132 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.132 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.132 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_s5CCY0lZsRXaG97sebP-L5L", 2020-11-29T16:11:11.132 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.133 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.133 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.133 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.133 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.133 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.134 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.134 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.134 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.333843_222.1", 2020-11-29T16:11:11.134 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.134 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-39:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39", 2020-11-29T16:11:11.135 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.333843820Z", 2020-11-29T16:11:11.135 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.135 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.135 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.135 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.136 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.136 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.136 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5H0pP9yCjtoMl1W4TvUgcb8", 2020-11-29T16:11:11.136 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.136 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.137 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.137 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.137 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.137 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.137 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.138 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.138 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.338434_223.1", 2020-11-29T16:11:11.138 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.138 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-39:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39", 2020-11-29T16:11:11.138 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.338434096Z", 2020-11-29T16:11:11.139 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.139 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.139 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.139 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.139 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.140 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.140 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5H0pP9yCjtoMl1W4TvUgcb8", 2020-11-29T16:11:11.140 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.140 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.141 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.141 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.141 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.141 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.141 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.142 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.142 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.340224_224.1", 2020-11-29T16:11:11.142 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.142 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-39", 2020-11-29T16:11:11.142 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.340224571Z", 2020-11-29T16:11:11.143 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.143 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.143 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.143 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.143 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.144 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.144 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Pp8cXl7djO1gxLhMq2AZXJp", 2020-11-29T16:11:11.144 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.144 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.144 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.145 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.145 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.145 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.145 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.145 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.146 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.343737_225.1", 2020-11-29T16:11:11.146 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.146 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-39", 2020-11-29T16:11:11.146 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.343737416Z", 2020-11-29T16:11:11.146 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.147 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.147 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.147 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.147 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.147 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.148 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Pp8cXl7djO1gxLhMq2AZXJp", 2020-11-29T16:11:11.148 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.148 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.148 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.148 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.149 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.149 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.149 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.149 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.149 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.368575_226.1", 2020-11-29T16:11:11.150 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.150 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-40:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40", 2020-11-29T16:11:11.150 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.368575114Z", 2020-11-29T16:11:11.150 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.151 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.151 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.151 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.151 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.151 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.152 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jgn-ZwFyVxv-tmED-Ggd4Hw", 2020-11-29T16:11:11.152 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.152 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.152 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.152 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.153 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.153 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.153 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.153 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.153 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.374510_227.1", 2020-11-29T16:11:11.154 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.154 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-40:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40", 2020-11-29T16:11:11.154 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.374510224Z", 2020-11-29T16:11:11.154 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.154 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.155 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.155 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.155 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.155 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.155 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jgn-ZwFyVxv-tmED-Ggd4Hw", 2020-11-29T16:11:11.156 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.156 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.156 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.156 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.156 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.157 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.157 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.157 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.157 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.376835_228.1", 2020-11-29T16:11:11.157 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.158 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-40", 2020-11-29T16:11:11.158 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.376835132Z", 2020-11-29T16:11:11.158 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.158 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.158 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.159 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.159 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.159 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.159 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_qrn1nKXKg0aZIU2cqKpKgju", 2020-11-29T16:11:11.160 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.160 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.160 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.160 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.160 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.161 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.161 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.161 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.161 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.380733_229.1", 2020-11-29T16:11:11.161 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.162 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-40", 2020-11-29T16:11:11.162 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.380733954Z", 2020-11-29T16:11:11.162 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.162 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.162 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.163 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.163 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.163 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.163 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_qrn1nKXKg0aZIU2cqKpKgju", 2020-11-29T16:11:11.163 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.164 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.164 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.164 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.164 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.164 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.165 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.165 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.165 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.430813_231.1", 2020-11-29T16:11:11.165 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.165 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-41:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41", 2020-11-29T16:11:11.166 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.430813301Z", 2020-11-29T16:11:11.166 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.166 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.166 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.166 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.167 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.167 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.167 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_upM-2-zq2ujNZABi5Lo2I6j", 2020-11-29T16:11:11.167 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.168 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.168 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.168 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.168 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.168 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.169 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.169 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.169 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.440999_232.1", 2020-11-29T16:11:11.169 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.169 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-41:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41", 2020-11-29T16:11:11.170 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.440999402Z", 2020-11-29T16:11:11.170 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.170 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.170 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.170 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.171 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.171 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.171 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_upM-2-zq2ujNZABi5Lo2I6j", 2020-11-29T16:11:11.171 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.171 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.172 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.172 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.172 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.172 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.172 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.173 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.173 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.444219_234.1", 2020-11-29T16:11:11.173 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.173 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-41", 2020-11-29T16:11:11.173 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.444219056Z", 2020-11-29T16:11:11.174 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.174 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.174 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.174 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.174 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.175 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.175 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_iwxqSbv55pvq-WwQMf1oR3e", 2020-11-29T16:11:11.175 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.175 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.175 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.176 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:11.176 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.176 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.176 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.177 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:11.177 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.452875_235.1", 2020-11-29T16:11:11.177 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:11.177 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-41", 2020-11-29T16:11:11.177 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.452875466Z", 2020-11-29T16:11:11.178 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:11.178 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:11.178 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:11.178 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:11.178 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.179 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:11.179 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_iwxqSbv55pvq-WwQMf1oR3e", 2020-11-29T16:11:11.179 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:11.179 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:11.179 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:11.180 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:11.180 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.180 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.180 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:11.180 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:11.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:11.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:11.185 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e24cddf9-396c-4261-a7bc-b4e1e678325e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:11.224 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:11.225 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:11.455 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:11.455 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.456 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665466.760427_46.1", 2020-11-29T16:11:11.456 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "user", 2020-11-29T16:11:11.456 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "rgw-multisite-test-user", 2020-11-29T16:11:11.456 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:57:46.760427145Z", 2020-11-29T16:11:11.456 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.457 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.457 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7jrnGFazjfCMV9OgOp-5ICR", 2020-11-29T16:11:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665466.770600_48.1", 2020-11-29T16:11:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "user", 2020-11-29T16:11:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "rgw-multisite-test-user", 2020-11-29T16:11:11.460 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:57:46.770600943Z", 2020-11-29T16:11:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.461 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7jrnGFazjfCMV9OgOp-5ICR", 2020-11-29T16:11:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.309764_49.1", 2020-11-29T16:11:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.464 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-1:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1", 2020-11-29T16:11:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.309764005Z", 2020-11-29T16:11:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.465 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tDx7cRFyPylOsARNJShhjdi", 2020-11-29T16:11:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.329195_52.1", 2020-11-29T16:11:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.468 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-1:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1", 2020-11-29T16:11:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.329195289Z", 2020-11-29T16:11:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.469 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tDx7cRFyPylOsARNJShhjdi", 2020-11-29T16:11:11.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.330857_53.1", 2020-11-29T16:11:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.472 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-1", 2020-11-29T16:11:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.330857588Z", 2020-11-29T16:11:11.473 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_da31gfzBeEr_qTz8sD5Z95W", 2020-11-29T16:11:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.334069_54.1", 2020-11-29T16:11:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-1", 2020-11-29T16:11:11.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.334069131Z", 2020-11-29T16:11:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.479 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.480 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_da31gfzBeEr_qTz8sD5Z95W", 2020-11-29T16:11:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.361540_55.1", 2020-11-29T16:11:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-2:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2", 2020-11-29T16:11:11.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.361540407Z", 2020-11-29T16:11:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.484 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.485 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jcxJvuSk8NgQdLMU8LRCoaP", 2020-11-29T16:11:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.486 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.366215_56.1", 2020-11-29T16:11:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-2:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2", 2020-11-29T16:11:11.488 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.366215003Z", 2020-11-29T16:11:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.489 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jcxJvuSk8NgQdLMU8LRCoaP", 2020-11-29T16:11:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.491 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.369096_57.1", 2020-11-29T16:11:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-2", 2020-11-29T16:11:11.493 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.369096676Z", 2020-11-29T16:11:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.494 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_F0nqKIPOuZ20dNAraH9mMFO", 2020-11-29T16:11:11.495 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.497 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.374355_58.1", 2020-11-29T16:11:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.498 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-2", 2020-11-29T16:11:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.374355442Z", 2020-11-29T16:11:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.499 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.500 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_F0nqKIPOuZ20dNAraH9mMFO", 2020-11-29T16:11:11.501 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.501 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.501 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665593.914359_59.1", 2020-11-29T16:11:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-3:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3", 2020-11-29T16:11:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.914359282Z", 2020-11-29T16:11:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.504 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_nPQIea0SAu2YbHSwS1hqdTS", 2020-11-29T16:11:11.505 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.506 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.507 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665593.919600_60.1", 2020-11-29T16:11:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-3:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3", 2020-11-29T16:11:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.919600334Z", 2020-11-29T16:11:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.509 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.510 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_nPQIea0SAu2YbHSwS1hqdTS", 2020-11-29T16:11:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.511 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.512 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665593.924502_61.1", 2020-11-29T16:11:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-3", 2020-11-29T16:11:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.924502922Z", 2020-11-29T16:11:11.513 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hgxnPf55gFvOJXZoH0wMw6D", 2020-11-29T16:11:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.515 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665593.929811_62.1", 2020-11-29T16:11:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-3", 2020-11-29T16:11:11.517 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.929811866Z", 2020-11-29T16:11:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.518 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hgxnPf55gFvOJXZoH0wMw6D", 2020-11-29T16:11:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.255794_63.1", 2020-11-29T16:11:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.521 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-4:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4", 2020-11-29T16:11:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.255794146Z", 2020-11-29T16:11:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.522 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dI-T3JKYYOksP3k9JHqk_4S", 2020-11-29T16:11:11.523 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.524 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.260821_65.1", 2020-11-29T16:11:11.525 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-4:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4", 2020-11-29T16:11:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.260821437Z", 2020-11-29T16:11:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.526 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.527 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dI-T3JKYYOksP3k9JHqk_4S", 2020-11-29T16:11:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.528 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.529 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.262726_66.1", 2020-11-29T16:11:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-4", 2020-11-29T16:11:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.262726231Z", 2020-11-29T16:11:11.530 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.531 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_H0hQB2qGLgINO4xvzkf-CBV", 2020-11-29T16:11:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.532 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.533 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.267845_67.1", 2020-11-29T16:11:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-4", 2020-11-29T16:11:11.534 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.267845461Z", 2020-11-29T16:11:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_H0hQB2qGLgINO4xvzkf-CBV", 2020-11-29T16:11:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.291993_68.1", 2020-11-29T16:11:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.538 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-5:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5", 2020-11-29T16:11:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.291993232Z", 2020-11-29T16:11:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.539 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_eNmUNSw8O2Q1UqGf3-a8M0y", 2020-11-29T16:11:11.540 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.541 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.297549_69.1", 2020-11-29T16:11:11.542 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-5:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5", 2020-11-29T16:11:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.297549197Z", 2020-11-29T16:11:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.543 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.544 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_eNmUNSw8O2Q1UqGf3-a8M0y", 2020-11-29T16:11:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.299887_70.1", 2020-11-29T16:11:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-5", 2020-11-29T16:11:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.299887736Z", 2020-11-29T16:11:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_axHo0yOC2VpaRAyEFVHuReN", 2020-11-29T16:11:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.307514_71.1", 2020-11-29T16:11:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-5", 2020-11-29T16:11:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.307514021Z", 2020-11-29T16:11:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_axHo0yOC2VpaRAyEFVHuReN", 2020-11-29T16:11:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.352257_72.1", 2020-11-29T16:11:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-6:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6", 2020-11-29T16:11:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.352257512Z", 2020-11-29T16:11:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.556 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_nbxbaUi2Y7VYrWoa72raC5j", 2020-11-29T16:11:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.558 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.559 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.359127_73.1", 2020-11-29T16:11:11.559 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.559 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-6:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6", 2020-11-29T16:11:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.359127428Z", 2020-11-29T16:11:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.560 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_nbxbaUi2Y7VYrWoa72raC5j", 2020-11-29T16:11:11.561 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.361298_74.1", 2020-11-29T16:11:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-6", 2020-11-29T16:11:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.361298888Z", 2020-11-29T16:11:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.565 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5-Sqi-mEXlqXjQFGM13_N21", 2020-11-29T16:11:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.365399_75.1", 2020-11-29T16:11:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-6", 2020-11-29T16:11:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.365399707Z", 2020-11-29T16:11:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5-Sqi-mEXlqXjQFGM13_N21", 2020-11-29T16:11:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.250797_76.1", 2020-11-29T16:11:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7", 2020-11-29T16:11:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.250797689Z", 2020-11-29T16:11:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_T6kPv-3jveF29kZeCLJlo-6", 2020-11-29T16:11:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.255494_79.1", 2020-11-29T16:11:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7", 2020-11-29T16:11:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.255494488Z", 2020-11-29T16:11:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_T6kPv-3jveF29kZeCLJlo-6", 2020-11-29T16:11:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.257909_80.1", 2020-11-29T16:11:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7", 2020-11-29T16:11:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.257909992Z", 2020-11-29T16:11:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.582 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.583 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.265880_81.1", 2020-11-29T16:11:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7", 2020-11-29T16:11:11.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.265880801Z", 2020-11-29T16:11:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.284752_82.1", 2020-11-29T16:11:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8", 2020-11-29T16:11:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.284752329Z", 2020-11-29T16:11:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Qg8VBYmtvIuXfPsbR8FJmF5", 2020-11-29T16:11:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.593 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.290062_83.1", 2020-11-29T16:11:11.593 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.594 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8", 2020-11-29T16:11:11.594 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.290062502Z", 2020-11-29T16:11:11.594 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.594 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.594 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.595 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.595 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.595 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Qg8VBYmtvIuXfPsbR8FJmF5", 2020-11-29T16:11:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.597 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.597 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.292190_84.1", 2020-11-29T16:11:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8", 2020-11-29T16:11:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.292190927Z", 2020-11-29T16:11:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.599 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.599 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.599 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.600 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:11.600 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.600 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.600 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.601 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.296685_85.1", 2020-11-29T16:11:11.602 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.602 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8", 2020-11-29T16:11:11.602 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.296685079Z", 2020-11-29T16:11:11.602 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.603 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.603 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.603 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.603 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.604 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:11.604 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.604 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.605 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.606 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.338804_86.1", 2020-11-29T16:11:11.606 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.606 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9", 2020-11-29T16:11:11.606 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.338804685Z", 2020-11-29T16:11:11.607 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.607 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.607 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.608 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.608 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_x_puG7-2t3lKG0zWJ2e3lY4", 2020-11-29T16:11:11.608 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.609 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.609 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.609 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.609 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.610 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.345806_87.1", 2020-11-29T16:11:11.610 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.610 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9", 2020-11-29T16:11:11.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.345806856Z", 2020-11-29T16:11:11.611 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.611 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.611 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.611 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.612 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.612 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_x_puG7-2t3lKG0zWJ2e3lY4", 2020-11-29T16:11:11.612 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.613 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.613 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.614 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.348268_88.1", 2020-11-29T16:11:11.614 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.615 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9", 2020-11-29T16:11:11.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.348268690Z", 2020-11-29T16:11:11.615 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.615 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.615 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.616 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.616 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.616 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:11.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.617 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.617 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.618 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.354068_89.1", 2020-11-29T16:11:11.619 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.619 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9", 2020-11-29T16:11:11.619 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.354068281Z", 2020-11-29T16:11:11.619 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.619 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.620 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.620 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.620 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.621 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:11.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.621 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.621 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.623 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.682590_90.1", 2020-11-29T16:11:11.623 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.623 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7", 2020-11-29T16:11:11.623 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.682590411Z", 2020-11-29T16:11:11.623 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.624 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.624 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:11.624 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.624 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.625 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:11.625 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.625 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.626 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-11-29T16:11:11.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.627 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.689390_92.1", 2020-11-29T16:11:11.627 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.627 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7", 2020-11-29T16:11:11.627 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.689390648Z", 2020-11-29T16:11:11.628 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.628 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.628 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:11.628 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.629 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.629 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:11.629 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.629 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.633 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.634 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.764882_93.1", 2020-11-29T16:11:11.634 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.634 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8", 2020-11-29T16:11:11.635 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.764882801Z", 2020-11-29T16:11:11.635 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.635 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.635 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:11.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.636 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.636 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:11.636 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.637 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.637 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-11-29T16:11:11.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.638 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.638 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.770654_94.1", 2020-11-29T16:11:11.638 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.639 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8", 2020-11-29T16:11:11.639 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.770654401Z", 2020-11-29T16:11:11.639 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.639 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.640 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:11.640 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.640 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.640 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:11.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.641 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.641 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.642 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.642 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.889388_95.1", 2020-11-29T16:11:11.643 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.643 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9", 2020-11-29T16:11:11.643 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.889388291Z", 2020-11-29T16:11:11.643 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.644 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.644 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:11.644 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.645 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.645 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:11.645 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.645 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.646 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-11-29T16:11:11.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.647 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.895400_96.1", 2020-11-29T16:11:11.647 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.647 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9", 2020-11-29T16:11:11.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400726Z", 2020-11-29T16:11:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.650 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.650 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.651 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.191369_34.1", 2020-11-29T16:11:11.651 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.652 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10", 2020-11-29T16:11:11.652 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.191369861Z", 2020-11-29T16:11:11.652 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.652 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.652 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YrVXr7ERmibtK1P81HX79vZ", 2020-11-29T16:11:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.195873_35.1", 2020-11-29T16:11:11.656 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.656 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10", 2020-11-29T16:11:11.656 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.195873130Z", 2020-11-29T16:11:11.656 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.656 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.657 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.657 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.657 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YrVXr7ERmibtK1P81HX79vZ", 2020-11-29T16:11:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.198497_36.1", 2020-11-29T16:11:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-10", 2020-11-29T16:11:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.198497073Z", 2020-11-29T16:11:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DFiygE422CgLw5QBztkfzu-", 2020-11-29T16:11:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.204454_37.1", 2020-11-29T16:11:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-10", 2020-11-29T16:11:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.204454835Z", 2020-11-29T16:11:11.665 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.665 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.665 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.666 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.666 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DFiygE422CgLw5QBztkfzu-", 2020-11-29T16:11:11.666 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.667 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.667 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.668 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.229632_38.1", 2020-11-29T16:11:11.668 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.668 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11", 2020-11-29T16:11:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.229632159Z", 2020-11-29T16:11:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.669 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.670 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.670 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_20WS3iYLQJHPayxR2Iu8xxa", 2020-11-29T16:11:11.670 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.671 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.671 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.672 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.234831_39.1", 2020-11-29T16:11:11.672 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.673 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11", 2020-11-29T16:11:11.673 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.234831409Z", 2020-11-29T16:11:11.673 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.673 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.673 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.674 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.674 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.674 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_20WS3iYLQJHPayxR2Iu8xxa", 2020-11-29T16:11:11.675 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.675 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.675 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.676 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.676 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.237152_40.1", 2020-11-29T16:11:11.677 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.677 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-11", 2020-11-29T16:11:11.677 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.237152720Z", 2020-11-29T16:11:11.677 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.677 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.678 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.678 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.678 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.679 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0w2vqVYxXcNZ-dU0Fiwn2qb", 2020-11-29T16:11:11.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.679 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.680 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.681 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.241905_41.1", 2020-11-29T16:11:11.681 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.681 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-11", 2020-11-29T16:11:11.681 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.241905299Z", 2020-11-29T16:11:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.683 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.683 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0w2vqVYxXcNZ-dU0Fiwn2qb", 2020-11-29T16:11:11.683 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.684 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.684 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.684 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.684 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.685 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.311228_42.1", 2020-11-29T16:11:11.685 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.686 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12", 2020-11-29T16:11:11.686 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.311228153Z", 2020-11-29T16:11:11.686 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.686 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.687 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.687 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.687 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.688 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jrQUaYBzA_aLdZ7o_IQP6PE", 2020-11-29T16:11:11.688 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.688 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.689 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.690 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.320192_43.1", 2020-11-29T16:11:11.690 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.690 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12", 2020-11-29T16:11:11.690 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.320192323Z", 2020-11-29T16:11:11.691 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.691 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.691 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.691 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.692 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.692 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jrQUaYBzA_aLdZ7o_IQP6PE", 2020-11-29T16:11:11.692 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.693 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.693 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.693 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.693 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.694 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.328671_44.1", 2020-11-29T16:11:11.694 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.694 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-12", 2020-11-29T16:11:11.695 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.328671604Z", 2020-11-29T16:11:11.695 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.695 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.695 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.695 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.696 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.696 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5MnB5hm2ISAIj9JGdxV50H9", 2020-11-29T16:11:11.696 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.697 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.697 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.697 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.697 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.698 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.335813_45.1", 2020-11-29T16:11:11.698 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.699 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-12", 2020-11-29T16:11:11.699 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.335813349Z", 2020-11-29T16:11:11.699 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.699 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.699 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.700 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.700 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.700 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5MnB5hm2ISAIj9JGdxV50H9", 2020-11-29T16:11:11.700 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.701 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.701 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.702 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.841649_46.1", 2020-11-29T16:11:11.703 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.703 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13", 2020-11-29T16:11:11.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.841649100Z", 2020-11-29T16:11:11.703 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.703 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.704 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.704 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.704 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.704 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Kqyzq54nxbVSdNcuPFIuxtb", 2020-11-29T16:11:11.705 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.705 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.706 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.707 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.847258_47.1", 2020-11-29T16:11:11.707 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.707 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13", 2020-11-29T16:11:11.707 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.847258533Z", 2020-11-29T16:11:11.708 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.708 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.708 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.708 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.709 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.709 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Kqyzq54nxbVSdNcuPFIuxtb", 2020-11-29T16:11:11.709 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.710 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.710 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.710 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.710 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.711 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.849488_48.1", 2020-11-29T16:11:11.711 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.711 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-13", 2020-11-29T16:11:11.712 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.849488010Z", 2020-11-29T16:11:11.712 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.712 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.712 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rjVcqahbW1Rw3RINe760YzT", 2020-11-29T16:11:11.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.715 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.853552_49.1", 2020-11-29T16:11:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-13", 2020-11-29T16:11:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.853552156Z", 2020-11-29T16:11:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.716 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.717 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rjVcqahbW1Rw3RINe760YzT", 2020-11-29T16:11:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.718 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.720 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.876908_50.1", 2020-11-29T16:11:11.720 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.720 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14", 2020-11-29T16:11:11.720 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.876908996Z", 2020-11-29T16:11:11.720 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.721 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.721 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.721 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.722 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.722 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YVkRWUTasfPQtpHA7NklgCE", 2020-11-29T16:11:11.722 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.722 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.723 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.724 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.881320_51.1", 2020-11-29T16:11:11.724 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.724 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14", 2020-11-29T16:11:11.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.881320871Z", 2020-11-29T16:11:11.725 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.725 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.725 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.726 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.726 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YVkRWUTasfPQtpHA7NklgCE", 2020-11-29T16:11:11.726 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.727 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.727 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.883170_52.1", 2020-11-29T16:11:11.728 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.729 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-14", 2020-11-29T16:11:11.729 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.883170592Z", 2020-11-29T16:11:11.729 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.729 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.730 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.730 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.730 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.730 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.731 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AsuYPfubiijDBKRDJK6qqjt", 2020-11-29T16:11:11.731 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.731 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.732 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.733 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.887114_53.1", 2020-11-29T16:11:11.733 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.734 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-14", 2020-11-29T16:11:11.734 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.887114146Z", 2020-11-29T16:11:11.734 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.734 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.735 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.735 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.735 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.736 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AsuYPfubiijDBKRDJK6qqjt", 2020-11-29T16:11:11.736 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.737 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.737 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.739 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.932301_54.1", 2020-11-29T16:11:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.740 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15", 2020-11-29T16:11:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.932301379Z", 2020-11-29T16:11:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.741 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.742 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_x_VdKNwXX4kJXxzVcmjduF6", 2020-11-29T16:11:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.743 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.936160_55.1", 2020-11-29T16:11:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15", 2020-11-29T16:11:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.936160887Z", 2020-11-29T16:11:11.745 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.747 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.747 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_x_VdKNwXX4kJXxzVcmjduF6", 2020-11-29T16:11:11.747 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.747 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.748 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.749 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.937891_56.1", 2020-11-29T16:11:11.749 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.749 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-15", 2020-11-29T16:11:11.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.937891458Z", 2020-11-29T16:11:11.750 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.750 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.750 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.750 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.751 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.751 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_QoRko9LJwKUXdid0giAhASQ", 2020-11-29T16:11:11.751 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.752 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.752 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.753 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.941872_57.1", 2020-11-29T16:11:11.753 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.753 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-15", 2020-11-29T16:11:11.754 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.941872760Z", 2020-11-29T16:11:11.754 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.754 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.754 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.755 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.755 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.755 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_QoRko9LJwKUXdid0giAhASQ", 2020-11-29T16:11:11.755 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.756 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.756 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.757 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.057028_59.1", 2020-11-29T16:11:11.757 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.758 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:11.758 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.057028183Z", 2020-11-29T16:11:11.758 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.758 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.759 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.759 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.759 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:11.760 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.760 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.760 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.761 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.761 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.061373_60.1", 2020-11-29T16:11:11.762 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.762 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:11.762 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.061373004Z", 2020-11-29T16:11:11.762 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.763 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.763 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.763 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.763 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.764 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:11.764 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.764 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.764 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.766 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.063911_61.1", 2020-11-29T16:11:11.766 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.766 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16", 2020-11-29T16:11:11.766 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.063911936Z", 2020-11-29T16:11:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.768 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.768 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qWQvPnGcRTlfP8YPDDWPU7j", 2020-11-29T16:11:11.768 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.069446_62.1", 2020-11-29T16:11:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16", 2020-11-29T16:11:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.069446267Z", 2020-11-29T16:11:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qWQvPnGcRTlfP8YPDDWPU7j", 2020-11-29T16:11:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.097339_63.1", 2020-11-29T16:11:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.097339684Z", 2020-11-29T16:11:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.779 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.102188_64.1", 2020-11-29T16:11:11.779 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.779 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:11.779 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.102188782Z", 2020-11-29T16:11:11.779 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.780 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.780 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.780 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.781 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:11.781 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.781 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.782 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.783 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.104862_65.1", 2020-11-29T16:11:11.783 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.783 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17", 2020-11-29T16:11:11.783 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.104862905Z", 2020-11-29T16:11:11.784 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.784 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.784 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.784 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.785 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.785 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3YDKF2LZHEWfgfUO2EvjsgL", 2020-11-29T16:11:11.785 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.785 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.786 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.787 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.787 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.108626_66.1", 2020-11-29T16:11:11.787 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.787 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17", 2020-11-29T16:11:11.788 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.108626431Z", 2020-11-29T16:11:11.788 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.788 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.788 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.788 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.789 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.789 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3YDKF2LZHEWfgfUO2EvjsgL", 2020-11-29T16:11:11.789 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.790 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.790 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.791 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.150629_67.1", 2020-11-29T16:11:11.791 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.792 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:11.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.150629298Z", 2020-11-29T16:11:11.792 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.792 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.792 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.793 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.793 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.793 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:11.794 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.794 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.794 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.795 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.158738_68.1", 2020-11-29T16:11:11.796 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.796 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:11.796 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.158738425Z", 2020-11-29T16:11:11.796 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.796 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.797 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.797 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.797 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.798 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:11.798 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.798 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.798 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.800 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.160845_69.1", 2020-11-29T16:11:11.800 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.800 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18", 2020-11-29T16:11:11.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.160845058Z", 2020-11-29T16:11:11.800 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.801 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.801 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.802 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.802 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5xSDI0jeDBdQUr4kb2mMHId", 2020-11-29T16:11:11.802 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.803 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.803 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.803 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.803 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.804 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.164732_70.1", 2020-11-29T16:11:11.804 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.804 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18", 2020-11-29T16:11:11.805 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.164732376Z", 2020-11-29T16:11:11.805 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.805 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.805 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.805 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.806 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.806 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5xSDI0jeDBdQUr4kb2mMHId", 2020-11-29T16:11:11.806 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.807 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.807 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.808 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.187080_71.1", 2020-11-29T16:11:11.808 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.808 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:11.809 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.187080319Z", 2020-11-29T16:11:11.809 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.809 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.809 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:11.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.810 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.810 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:11.810 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.811 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.811 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.811 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.812 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.812 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.195335_72.1", 2020-11-29T16:11:11.812 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.813 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:11.813 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.195335470Z", 2020-11-29T16:11:11.813 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.813 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.814 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:11.814 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.814 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.814 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:11.815 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.815 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.815 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.816 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.203159_73.1", 2020-11-29T16:11:11.817 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.817 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:11.817 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.203159941Z", 2020-11-29T16:11:11.817 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.817 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.818 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:11.818 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.818 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.819 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:11.819 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.819 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.819 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.820 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.820 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.821 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.210320_74.1", 2020-11-29T16:11:11.821 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.821 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:11.821 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.210320129Z", 2020-11-29T16:11:11.821 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.822 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.822 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:11.822 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.823 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.823 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:11.823 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.823 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.824 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.825 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.221591_75.1", 2020-11-29T16:11:11.825 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.825 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:11.825 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.221591576Z", 2020-11-29T16:11:11.826 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.826 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.826 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:11.826 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.827 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.827 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:11.827 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.229493_76.1", 2020-11-29T16:11:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.829 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:11.830 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.229493483Z", 2020-11-29T16:11:11.830 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.830 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.830 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:11.831 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.831 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.831 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:11.831 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.967652_86.1", 2020-11-29T16:11:11.833 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.834 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:11.834 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.967652642Z", 2020-11-29T16:11:11.834 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.834 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.834 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.835 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.835 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.835 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:11.836 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.836 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.836 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.837 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.972279_87.1", 2020-11-29T16:11:11.838 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.838 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:11.838 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.972279097Z", 2020-11-29T16:11:11.838 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.839 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.839 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.839 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.839 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.840 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:11.840 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.840 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.841 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.842 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.974314_88.1", 2020-11-29T16:11:11.842 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.842 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19", 2020-11-29T16:11:11.842 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.974314737Z", 2020-11-29T16:11:11.843 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.843 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.843 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.843 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.844 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.844 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iGyiAtOMEjJAj1PjCXsPDZz", 2020-11-29T16:11:11.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.845 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.845 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.846 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.978564_89.1", 2020-11-29T16:11:11.846 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.846 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19", 2020-11-29T16:11:11.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.978564499Z", 2020-11-29T16:11:11.847 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.847 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.847 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.847 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.848 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.848 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iGyiAtOMEjJAj1PjCXsPDZz", 2020-11-29T16:11:11.848 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.849 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.849 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.850 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.984258_90.1", 2020-11-29T16:11:11.850 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.850 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:11.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.984258860Z", 2020-11-29T16:11:11.851 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.851 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.851 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:11.852 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.852 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.852 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:11.852 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.853 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.853 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.854 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.854 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.991097_91.1", 2020-11-29T16:11:11.854 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.855 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:11.855 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.991097909Z", 2020-11-29T16:11:11.855 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.855 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.856 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:11.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.856 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.856 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:11.857 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.858 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665997.215638_94.1", 2020-11-29T16:11:11.859 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.859 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:11.859 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.215638352Z", 2020-11-29T16:11:11.859 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.860 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.860 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.860 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.860 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.861 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.861 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.861 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.861 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.863 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665997.220413_95.1", 2020-11-29T16:11:11.863 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.863 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:11.863 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.220413426Z", 2020-11-29T16:11:11.864 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.864 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.864 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.864 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.865 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.865 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.865 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.866 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.866 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.867 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665997.222814_96.1", 2020-11-29T16:11:11.867 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.867 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20", 2020-11-29T16:11:11.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.222814425Z", 2020-11-29T16:11:11.868 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.868 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.868 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.869 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.869 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.869 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_caPUfpnBPovw7pnaLhfQKQu", 2020-11-29T16:11:11.870 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.870 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.870 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.875 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665997.230793_97.1", 2020-11-29T16:11:11.875 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.876 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20", 2020-11-29T16:11:11.876 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.230793956Z", 2020-11-29T16:11:11.876 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.876 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.877 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.877 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.877 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.877 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_caPUfpnBPovw7pnaLhfQKQu", 2020-11-29T16:11:11.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.878 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.878 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.878 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.879 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.879 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.879 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666001.819331_99.1", 2020-11-29T16:11:11.879 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.880 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:11.880 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:41.819331435Z", 2020-11-29T16:11:11.880 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.880 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.881 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.881 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.881 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.881 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.882 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.882 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.883 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666001.830749_100.1", 2020-11-29T16:11:11.883 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.884 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:11.884 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:41.830749663Z", 2020-11-29T16:11:11.884 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.884 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.885 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.885 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.886 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.886 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.886 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.886 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.887 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666061.809169_106.1", 2020-11-29T16:11:11.888 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.888 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:11.888 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:07:41.809169955Z", 2020-11-29T16:11:11.888 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.889 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.889 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.889 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.889 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.890 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:11.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.890 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.891 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.892 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666061.819824_107.1", 2020-11-29T16:11:11.892 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.892 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:11.892 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:07:41.819824059Z", 2020-11-29T16:11:11.892 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.893 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.893 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.893 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.893 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.894 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:11.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:11.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.896 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.732730_108.1", 2020-11-29T16:11:11.896 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.896 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:11.896 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.732730870Z", 2020-11-29T16:11:11.897 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.897 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.897 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.897 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.898 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.898 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:11.898 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.899 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.899 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.899 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.899 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.900 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.738936_109.1", 2020-11-29T16:11:11.900 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.900 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:11.900 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.738936839Z", 2020-11-29T16:11:11.901 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.901 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.901 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.902 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.902 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:11.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.902 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.903 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.904 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.740862_110.1", 2020-11-29T16:11:11.904 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.904 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21", 2020-11-29T16:11:11.904 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.740862527Z", 2020-11-29T16:11:11.904 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.905 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.905 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.905 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.905 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.906 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1LtU-sUHWZQfFCaHHia4XL6", 2020-11-29T16:11:11.906 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.906 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.906 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.907 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.745932_111.1", 2020-11-29T16:11:11.908 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.908 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21", 2020-11-29T16:11:11.908 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.745932979Z", 2020-11-29T16:11:11.908 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.908 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.909 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.909 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.909 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.909 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1LtU-sUHWZQfFCaHHia4XL6", 2020-11-29T16:11:11.910 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.910 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.910 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.911 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.911 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.771777_112.1", 2020-11-29T16:11:11.912 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.912 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:11.912 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.771777245Z", 2020-11-29T16:11:11.912 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.912 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.913 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.913 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.913 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:11.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.914 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.914 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.914 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.914 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.915 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.779214_113.1", 2020-11-29T16:11:11.915 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.915 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:11.916 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.779214800Z", 2020-11-29T16:11:11.916 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.916 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.916 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.916 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.917 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.917 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:11.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.918 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.918 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.919 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.781033_114.1", 2020-11-29T16:11:11.919 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.919 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22", 2020-11-29T16:11:11.919 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.781033305Z", 2020-11-29T16:11:11.920 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.920 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.920 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.921 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.921 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yAIfpIe3NidR6qhQ52Y6gQ9", 2020-11-29T16:11:11.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.921 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.922 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.923 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.786435_115.1", 2020-11-29T16:11:11.923 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.923 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22", 2020-11-29T16:11:11.923 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.786435846Z", 2020-11-29T16:11:11.923 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.924 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.924 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.924 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.924 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yAIfpIe3NidR6qhQ52Y6gQ9", 2020-11-29T16:11:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.925 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.926 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.833583_116.1", 2020-11-29T16:11:11.927 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.927 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:11.927 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.833583970Z", 2020-11-29T16:11:11.927 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.927 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.928 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.928 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.928 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.928 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:11.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.929 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.929 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.930 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.838249_117.1", 2020-11-29T16:11:11.931 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.931 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:11.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.838249602Z", 2020-11-29T16:11:11.931 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.931 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.932 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.932 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.932 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.932 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:11.933 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.933 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.933 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.934 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.840897_118.1", 2020-11-29T16:11:11.934 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23", 2020-11-29T16:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.840897485Z", 2020-11-29T16:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.935 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yUDBs52ibfejiizLQQANf-x", 2020-11-29T16:11:11.936 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.937 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.937 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.937 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.937 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.938 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.846339_119.1", 2020-11-29T16:11:11.938 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.938 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23", 2020-11-29T16:11:11.939 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.846339269Z", 2020-11-29T16:11:11.939 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.939 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.939 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.940 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.940 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yUDBs52ibfejiizLQQANf-x", 2020-11-29T16:11:11.940 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.941 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.941 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.942 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.869230_120.1", 2020-11-29T16:11:11.942 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.942 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:11.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.869230022Z", 2020-11-29T16:11:11.943 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.943 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.943 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:11.943 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.944 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.944 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:11.944 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.944 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.945 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.946 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.875173_121.1", 2020-11-29T16:11:11.946 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.946 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:11.946 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.875173699Z", 2020-11-29T16:11:11.946 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.947 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.947 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:11.947 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.947 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.948 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:11.948 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.948 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.948 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.949 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.879784_122.1", 2020-11-29T16:11:11.950 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.950 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:11.950 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.879784537Z", 2020-11-29T16:11:11.950 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.950 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.951 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:11.951 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.951 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.952 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:11.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.952 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.952 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.953 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.886953_123.1", 2020-11-29T16:11:11.954 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.954 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:11.954 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.886953932Z", 2020-11-29T16:11:11.954 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.954 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.955 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:11.955 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.955 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.955 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:11.956 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.956 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.956 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.957 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.901158_124.1", 2020-11-29T16:11:11.958 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.958 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:11.958 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.901158656Z", 2020-11-29T16:11:11.958 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.958 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.959 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:11.959 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.959 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.959 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:11.960 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.960 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.960 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.961 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.915755_125.1", 2020-11-29T16:11:11.962 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.962 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:11.962 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.915755723Z", 2020-11-29T16:11:11.962 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.962 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.963 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:11.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.963 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.963 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:11.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:11.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.964 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.964 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.965 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.456443_126.1", 2020-11-29T16:11:11.965 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.966 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:11.966 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.456443611Z", 2020-11-29T16:11:11.966 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.966 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.967 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.967 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.967 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.968 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:11.968 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.968 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.968 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.969 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.460387_127.1", 2020-11-29T16:11:11.970 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.970 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:11.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.460387996Z", 2020-11-29T16:11:11.970 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.970 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.971 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.971 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.971 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.971 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:11.972 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.972 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.972 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.973 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.462193_128.1", 2020-11-29T16:11:11.973 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.974 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24", 2020-11-29T16:11:11.974 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.462193185Z", 2020-11-29T16:11:11.974 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.974 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.974 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.975 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.975 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_8rFe9Ex-HuV_dw_B2ecHBhB", 2020-11-29T16:11:11.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.976 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.976 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.977 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.977 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.467714_129.1", 2020-11-29T16:11:11.977 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.977 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24", 2020-11-29T16:11:11.978 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.467714061Z", 2020-11-29T16:11:11.978 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.978 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.978 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.978 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.979 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.979 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_8rFe9Ex-HuV_dw_B2ecHBhB", 2020-11-29T16:11:11.979 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.980 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.980 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.981 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.490669_130.1", 2020-11-29T16:11:11.981 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.981 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:11.981 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.490669098Z", 2020-11-29T16:11:11.982 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.982 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.982 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.982 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.983 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.983 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:11.983 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.983 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.984 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.984 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.496194_131.1", 2020-11-29T16:11:11.985 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.985 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:11.985 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.496194848Z", 2020-11-29T16:11:11.985 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.985 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.986 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.986 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.986 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.986 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.987 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.988 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.988 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.498339_132.1", 2020-11-29T16:11:11.988 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.989 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25", 2020-11-29T16:11:11.989 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.498339597Z", 2020-11-29T16:11:11.989 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.989 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.989 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.990 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.990 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.990 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s63DKGRwg1GnITE9VweheW4", 2020-11-29T16:11:11.990 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.991 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.991 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.991 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.991 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.992 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.502532_133.1", 2020-11-29T16:11:11.992 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:11.992 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25", 2020-11-29T16:11:11.993 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.502532461Z", 2020-11-29T16:11:11.993 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.993 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.993 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.994 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.994 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s63DKGRwg1GnITE9VweheW4", 2020-11-29T16:11:11.994 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.995 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.995 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:11.995 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.995 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:11.996 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.557238_134.1", 2020-11-29T16:11:11.996 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:11.996 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:11.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.557238717Z", 2020-11-29T16:11:11.997 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:11.997 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:11.997 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:11.997 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:11.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.998 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:11.998 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:11.998 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:11.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:11.999 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:11.999 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:11.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:11.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.000 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.562930_135.1", 2020-11-29T16:11:12.000 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.000 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:12.000 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.562930060Z", 2020-11-29T16:11:12.001 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.001 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.001 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.002 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.002 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.002 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.002 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.003 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.003 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.003 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.004 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.565930_136.1", 2020-11-29T16:11:12.004 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.004 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26", 2020-11-29T16:11:12.004 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.565930147Z", 2020-11-29T16:11:12.004 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.005 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HEKlnFCl4gKqxl9FXC6_Slr", 2020-11-29T16:11:12.006 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.006 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.006 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.006 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.007 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.574718_137.1", 2020-11-29T16:11:12.008 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.008 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26", 2020-11-29T16:11:12.008 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.574718238Z", 2020-11-29T16:11:12.008 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.008 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.009 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.009 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.009 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.009 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HEKlnFCl4gKqxl9FXC6_Slr", 2020-11-29T16:11:12.010 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.011 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.011 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.594772_138.1", 2020-11-29T16:11:12.011 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.012 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:12.012 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.594772332Z", 2020-11-29T16:11:12.012 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.012 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.012 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:12.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.013 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.013 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:12.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.014 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.014 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.015 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.599622_139.1", 2020-11-29T16:11:12.015 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.015 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:12.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.599622375Z", 2020-11-29T16:11:12.016 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.016 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.016 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:12.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.017 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.017 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:12.017 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.019 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.603089_140.1", 2020-11-29T16:11:12.019 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.019 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:12.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.603089353Z", 2020-11-29T16:11:12.020 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.020 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.020 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:12.020 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.021 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.021 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:12.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:12.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.021 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.022 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.023 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.612024_141.1", 2020-11-29T16:11:12.023 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.023 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:12.023 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.612024636Z", 2020-11-29T16:11:12.024 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.024 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.024 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:12.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.025 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.025 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:12.025 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:12.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.025 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.026 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.026 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.026 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.027 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.616582_142.1", 2020-11-29T16:11:12.027 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.027 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:12.027 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.616582099Z", 2020-11-29T16:11:12.028 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.028 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.028 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:12.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.029 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.029 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:12.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.030 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.030 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.031 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.624065_143.1", 2020-11-29T16:11:12.031 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.031 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:12.031 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.624065742Z", 2020-11-29T16:11:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.033 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.033 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:12.033 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.633769_144.1", 2020-11-29T16:11:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:12.035 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.633769558Z", 2020-11-29T16:11:12.036 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.036 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.036 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:12.036 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.037 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.037 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:12.037 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:12.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.037 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.038 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.039 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.641875_145.1", 2020-11-29T16:11:12.039 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.039 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:12.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.641875589Z", 2020-11-29T16:11:12.039 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.040 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.040 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:12.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.040 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.040 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:12.041 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:12.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.041 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.041 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.041 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.042 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.042 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.655938_146.1", 2020-11-29T16:11:12.042 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.043 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:12.043 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.655938546Z", 2020-11-29T16:11:12.043 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.043 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.043 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.044 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.044 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.044 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.044 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.045 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.045 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.045 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.045 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.046 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.665066_147.1", 2020-11-29T16:11:12.046 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.046 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:12.047 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.665066109Z", 2020-11-29T16:11:12.047 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.047 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.047 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.048 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.048 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.048 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.048 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.049 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.050 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.677379_148.1", 2020-11-29T16:11:12.050 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.050 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:12.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.677379068Z", 2020-11-29T16:11:12.050 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.051 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.051 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.051 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.052 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.052 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:12.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.052 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.052 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.053 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.684234_149.1", 2020-11-29T16:11:12.054 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.054 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:12.054 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.684234315Z", 2020-11-29T16:11:12.054 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.054 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.055 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.055 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.055 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:12.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:12.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.056 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.056 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.057 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.338993_154.1", 2020-11-29T16:11:12.057 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.058 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:12.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.338993057Z", 2020-11-29T16:11:12.058 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.058 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.058 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.059 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.059 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.059 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:12.059 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.061 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.346126_155.1", 2020-11-29T16:11:12.061 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.061 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:12.062 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.346126383Z", 2020-11-29T16:11:12.062 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.062 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.062 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.062 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.063 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.063 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:12.063 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.063 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.064 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.065 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.348222_156.1", 2020-11-29T16:11:12.065 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.065 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27", 2020-11-29T16:11:12.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.348222396Z", 2020-11-29T16:11:12.065 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.066 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.066 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.066 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.066 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.067 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Wc4auwc30IepX657G2eZM-u", 2020-11-29T16:11:12.067 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.067 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.067 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.068 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.353951_157.1", 2020-11-29T16:11:12.069 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.069 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27", 2020-11-29T16:11:12.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.353951918Z", 2020-11-29T16:11:12.069 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.069 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.070 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.070 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.070 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.070 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Wc4auwc30IepX657G2eZM-u", 2020-11-29T16:11:12.071 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.071 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.071 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.072 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.390106_158.1", 2020-11-29T16:11:12.072 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.073 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:12.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.390106865Z", 2020-11-29T16:11:12.073 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.073 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.073 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.074 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.074 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:12.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.075 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.075 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.076 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.397881_159.1", 2020-11-29T16:11:12.076 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.076 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:12.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.397881409Z", 2020-11-29T16:11:12.077 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.077 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.077 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.077 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.078 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.078 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:12.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.078 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.079 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.080 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.400119_160.1", 2020-11-29T16:11:12.080 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.080 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28", 2020-11-29T16:11:12.080 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.400119169Z", 2020-11-29T16:11:12.081 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.081 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.081 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.081 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.081 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.082 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vorB3p52YuzxIbSF-GHscIz", 2020-11-29T16:11:12.082 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.082 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.082 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.083 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.083 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.404155_161.1", 2020-11-29T16:11:12.084 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.084 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28", 2020-11-29T16:11:12.084 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.404155690Z", 2020-11-29T16:11:12.084 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.084 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.085 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.085 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.085 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.085 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vorB3p52YuzxIbSF-GHscIz", 2020-11-29T16:11:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.087 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.461726_162.1", 2020-11-29T16:11:12.087 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.461726137Z", 2020-11-29T16:11:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.090 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.090 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.091 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.466239_163.1", 2020-11-29T16:11:12.091 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.091 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.466239727Z", 2020-11-29T16:11:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.093 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.093 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:12.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.095 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.471274_164.1", 2020-11-29T16:11:12.095 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.095 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29", 2020-11-29T16:11:12.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.471274792Z", 2020-11-29T16:11:12.095 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.096 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.096 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.096 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.096 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.097 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tW9vOLbEcF0UPbuJYTP9PYC", 2020-11-29T16:11:12.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.097 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.097 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.098 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.475990_165.1", 2020-11-29T16:11:12.098 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.099 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29", 2020-11-29T16:11:12.099 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.475990005Z", 2020-11-29T16:11:12.099 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.099 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.099 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.100 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.100 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.100 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.100 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tW9vOLbEcF0UPbuJYTP9PYC", 2020-11-29T16:11:12.100 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.500752_166.1", 2020-11-29T16:11:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.102 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.500752451Z", 2020-11-29T16:11:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:12.103 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.105 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.105 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.105 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.105 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.105 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.106 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.106 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.506749_167.1", 2020-11-29T16:11:12.106 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.106 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:12.106 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.506749250Z", 2020-11-29T16:11:12.107 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.107 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.107 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:12.107 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.108 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.108 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:12.108 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.108 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.109 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.109 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.109 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.110 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.561039_168.1", 2020-11-29T16:11:12.110 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.110 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:12.110 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.561039789Z", 2020-11-29T16:11:12.111 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.111 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.111 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:12.111 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.112 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.112 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:12.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.112 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.113 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.113 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.113 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.113 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.114 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.568104_169.1", 2020-11-29T16:11:12.114 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.114 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:12.114 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.568104840Z", 2020-11-29T16:11:12.114 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.115 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.115 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:12.115 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.115 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.116 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.117 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.624595_170.1", 2020-11-29T16:11:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.624595921Z", 2020-11-29T16:11:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.118 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.119 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.631524_171.1", 2020-11-29T16:11:12.121 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.631524395Z", 2020-11-29T16:11:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.122 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:12.123 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.123 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.123 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:12.123 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.124 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.124 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.125 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.709256_172.1", 2020-11-29T16:11:12.125 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.125 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:12.126 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.709256940Z", 2020-11-29T16:11:12.126 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.126 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.126 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.126 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.127 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.127 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:12.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.128 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.128 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.129 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.129 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.713773_173.1", 2020-11-29T16:11:12.129 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.129 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:12.129 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.713773728Z", 2020-11-29T16:11:12.130 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.130 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.130 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.130 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.131 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.131 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:12.131 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.131 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.132 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.132 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.133 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.718379_174.1", 2020-11-29T16:11:12.133 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.133 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30", 2020-11-29T16:11:12.133 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.718379824Z", 2020-11-29T16:11:12.133 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.134 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.134 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.134 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.135 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h_WBzhTZe664PHorKv6_Am0", 2020-11-29T16:11:12.135 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.135 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.135 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.136 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.136 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.723686_175.1", 2020-11-29T16:11:12.137 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.137 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30", 2020-11-29T16:11:12.137 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.723686356Z", 2020-11-29T16:11:12.137 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.137 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.138 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.138 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.138 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.138 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h_WBzhTZe664PHorKv6_Am0", 2020-11-29T16:11:12.139 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.139 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.139 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.139 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.139 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.140 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.140 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.752571_176.1", 2020-11-29T16:11:12.140 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.141 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:12.141 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.752571930Z", 2020-11-29T16:11:12.141 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.141 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.141 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.141 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.142 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.142 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:12.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.143 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.143 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.144 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.756904_177.1", 2020-11-29T16:11:12.144 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.144 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:12.144 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.756904681Z", 2020-11-29T16:11:12.145 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.145 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.145 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.145 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.145 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.146 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:12.146 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.146 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.146 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.147 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.147 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.147 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.147 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.759821_178.1", 2020-11-29T16:11:12.148 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.148 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31", 2020-11-29T16:11:12.148 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.759821922Z", 2020-11-29T16:11:12.148 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.148 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.149 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.149 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.149 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.149 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_eKjXcKGmg5yyLHdihUsR2sW", 2020-11-29T16:11:12.150 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.150 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.150 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.150 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.151 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.151 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.764291_179.1", 2020-11-29T16:11:12.151 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.152 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31", 2020-11-29T16:11:12.152 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.764291393Z", 2020-11-29T16:11:12.152 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.152 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.152 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_eKjXcKGmg5yyLHdihUsR2sW", 2020-11-29T16:11:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.825137_180.1", 2020-11-29T16:11:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.825137534Z", 2020-11-29T16:11:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.829684_181.1", 2020-11-29T16:11:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.829684939Z", 2020-11-29T16:11:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.831781_182.1", 2020-11-29T16:11:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32", 2020-11-29T16:11:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.831781072Z", 2020-11-29T16:11:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dhvj1N7nCxk5qukpVfy3IjW", 2020-11-29T16:11:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.835966_183.1", 2020-11-29T16:11:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32", 2020-11-29T16:11:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.835966939Z", 2020-11-29T16:11:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dhvj1N7nCxk5qukpVfy3IjW", 2020-11-29T16:11:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.169 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.169 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.170 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.170 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.860087_184.1", 2020-11-29T16:11:12.170 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.170 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:12.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.860087065Z", 2020-11-29T16:11:12.171 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.171 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.171 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:12.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.172 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.172 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:12.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.172 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.172 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.173 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.173 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.865870_185.1", 2020-11-29T16:11:12.174 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.174 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:12.174 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.865870483Z", 2020-11-29T16:11:12.174 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.175 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.175 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:12.175 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.175 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.176 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:12.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.176 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.176 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.176 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.177 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.914709_186.1", 2020-11-29T16:11:12.177 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.178 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:12.178 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.914709667Z", 2020-11-29T16:11:12.178 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.178 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.178 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:12.179 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.179 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.179 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:12.179 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.180 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.180 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.181 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.925154_187.1", 2020-11-29T16:11:12.181 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.181 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:12.182 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.925154339Z", 2020-11-29T16:11:12.182 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.182 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.182 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:12.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.183 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.183 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:12.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.184 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.184 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.185 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.991321_188.1", 2020-11-29T16:11:12.185 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.185 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:12.185 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.991321647Z", 2020-11-29T16:11:12.186 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.186 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.186 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:12.186 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.187 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.187 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:12.187 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.187 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.188 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.188 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.188 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.999983_189.1", 2020-11-29T16:11:12.189 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.189 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:12.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.999983004Z", 2020-11-29T16:11:12.189 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.189 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.190 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.076605_190.1", 2020-11-29T16:11:12.192 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.076605830Z", 2020-11-29T16:11:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.193 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.195 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.081952_191.1", 2020-11-29T16:11:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:12.196 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.081952008Z", 2020-11-29T16:11:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.198 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.198 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:12.198 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.198 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.199 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.199 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.199 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.199 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.200 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.083820_192.1", 2020-11-29T16:11:12.200 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.200 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33", 2020-11-29T16:11:12.200 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.083820333Z", 2020-11-29T16:11:12.200 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.201 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.201 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.201 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.202 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_aScNky5UrTuH12xWR79F83e", 2020-11-29T16:11:12.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.203 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.088982_193.1", 2020-11-29T16:11:12.203 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.204 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33", 2020-11-29T16:11:12.204 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.088982559Z", 2020-11-29T16:11:12.204 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.204 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.204 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.205 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.205 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_aScNky5UrTuH12xWR79F83e", 2020-11-29T16:11:12.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.121086_194.1", 2020-11-29T16:11:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.207 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.121086133Z", 2020-11-29T16:11:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.209 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.133205_195.1", 2020-11-29T16:11:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.133205341Z", 2020-11-29T16:11:12.211 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.212 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.212 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.212 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.213 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:12.213 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.213 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.213 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.214 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.214 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.214 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.138006_196.1", 2020-11-29T16:11:12.215 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.215 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34", 2020-11-29T16:11:12.215 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.138006291Z", 2020-11-29T16:11:12.215 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.215 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.215 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.216 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.216 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.216 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_9r_K5jlLAncl40ETfoVeJjK", 2020-11-29T16:11:12.216 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.153529_197.1", 2020-11-29T16:11:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34", 2020-11-29T16:11:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.153529708Z", 2020-11-29T16:11:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_9r_K5jlLAncl40ETfoVeJjK", 2020-11-29T16:11:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.225476_198.1", 2020-11-29T16:11:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.225476965Z", 2020-11-29T16:11:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.225 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.230332_199.1", 2020-11-29T16:11:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.230332153Z", 2020-11-29T16:11:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.232895_200.1", 2020-11-29T16:11:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35", 2020-11-29T16:11:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.232895662Z", 2020-11-29T16:11:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BauiNjhDLe4JD_SZUYzW5BH", 2020-11-29T16:11:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.237930_201.1", 2020-11-29T16:11:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35", 2020-11-29T16:11:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.237930233Z", 2020-11-29T16:11:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.235 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BauiNjhDLe4JD_SZUYzW5BH", 2020-11-29T16:11:12.235 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.235 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.235 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.236 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.261144_202.1", 2020-11-29T16:11:12.237 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.237 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:12.237 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.261144444Z", 2020-11-29T16:11:12.237 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.237 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.238 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:12.238 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.238 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.238 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:12.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.239 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.239 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.240 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.266462_203.1", 2020-11-29T16:11:12.240 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.240 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:12.241 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.266462524Z", 2020-11-29T16:11:12.241 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.241 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.241 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:12.241 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.242 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.242 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:12.242 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.243 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.243 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.244 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666124.313715_204.1", 2020-11-29T16:11:12.244 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.244 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:12.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:44.313715914Z", 2020-11-29T16:11:12.244 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.245 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.245 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:12.245 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.245 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.246 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:12.246 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.246 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.246 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.247 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666124.327834_205.1", 2020-11-29T16:11:12.247 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.248 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:12.248 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:44.327834836Z", 2020-11-29T16:11:12.248 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.248 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.248 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:12.249 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.249 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.249 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:12.249 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.250 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.250 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.389734_206.1", 2020-11-29T16:11:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.251 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.389734131Z", 2020-11-29T16:11:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:12.252 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.253 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.253 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:12.253 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.254 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.254 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.255 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.255 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.399959_207.1", 2020-11-29T16:11:12.255 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.255 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:12.255 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.399959602Z", 2020-11-29T16:11:12.256 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.256 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.256 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:12.256 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.257 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.257 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:12.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:12.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.257 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.258 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.258 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.259 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.480704_208.1", 2020-11-29T16:11:12.259 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.259 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36", 2020-11-29T16:11:12.259 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.480704401Z", 2020-11-29T16:11:12.259 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.260 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.260 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.260 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.261 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_UJxOrACWaVFo5tZbkaJd5aB", 2020-11-29T16:11:12.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.262 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.262 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.485309_209.1", 2020-11-29T16:11:12.263 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.263 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36", 2020-11-29T16:11:12.263 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.485309628Z", 2020-11-29T16:11:12.263 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.263 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.264 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.264 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.264 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.264 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_UJxOrACWaVFo5tZbkaJd5aB", 2020-11-29T16:11:12.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.265 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.265 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.487975_210.1", 2020-11-29T16:11:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-36", 2020-11-29T16:11:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.487975960Z", 2020-11-29T16:11:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.267 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.268 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.268 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VZJxPfFqHBrvBon1FcGTxPs", 2020-11-29T16:11:12.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.269 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.269 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.269 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.269 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.270 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.270 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.491713_211.1", 2020-11-29T16:11:12.270 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.270 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-36", 2020-11-29T16:11:12.271 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.491713665Z", 2020-11-29T16:11:12.271 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.271 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.271 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.271 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.272 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.272 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VZJxPfFqHBrvBon1FcGTxPs", 2020-11-29T16:11:12.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.272 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.273 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.274 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.525177_212.1", 2020-11-29T16:11:12.274 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.274 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37", 2020-11-29T16:11:12.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.525177090Z", 2020-11-29T16:11:12.274 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.275 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.275 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.275 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.275 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.276 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_syk3UA8fSzEvnh7yGHxS7iU", 2020-11-29T16:11:12.276 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.277 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.534079_213.1", 2020-11-29T16:11:12.277 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.278 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37", 2020-11-29T16:11:12.278 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.534079771Z", 2020-11-29T16:11:12.278 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.278 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.279 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.279 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.279 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_syk3UA8fSzEvnh7yGHxS7iU", 2020-11-29T16:11:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.540941_214.1", 2020-11-29T16:11:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-37", 2020-11-29T16:11:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.540941786Z", 2020-11-29T16:11:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_LPLM1eQYX53fxff8emhXqjO", 2020-11-29T16:11:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.547825_215.1", 2020-11-29T16:11:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-37", 2020-11-29T16:11:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.547825214Z", 2020-11-29T16:11:12.286 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.286 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.286 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.286 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_LPLM1eQYX53fxff8emhXqjO", 2020-11-29T16:11:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.590883_216.1", 2020-11-29T16:11:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38", 2020-11-29T16:11:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.590883464Z", 2020-11-29T16:11:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_sluiDwZnW0jlavok--xXUQM", 2020-11-29T16:11:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.595783_217.1", 2020-11-29T16:11:12.292 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38", 2020-11-29T16:11:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.595783563Z", 2020-11-29T16:11:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.293 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_sluiDwZnW0jlavok--xXUQM", 2020-11-29T16:11:12.294 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.597926_218.1", 2020-11-29T16:11:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.296 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-38", 2020-11-29T16:11:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.597926148Z", 2020-11-29T16:11:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s5CCY0lZsRXaG97sebP-L5L", 2020-11-29T16:11:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.602582_219.1", 2020-11-29T16:11:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-38", 2020-11-29T16:11:12.300 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.602582602Z", 2020-11-29T16:11:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s5CCY0lZsRXaG97sebP-L5L", 2020-11-29T16:11:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.302 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.333843_222.1", 2020-11-29T16:11:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-39:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39", 2020-11-29T16:11:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.333843820Z", 2020-11-29T16:11:12.304 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.305 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5H0pP9yCjtoMl1W4TvUgcb8", 2020-11-29T16:11:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.306 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.338434_223.1", 2020-11-29T16:11:12.307 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-39:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39", 2020-11-29T16:11:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.338434096Z", 2020-11-29T16:11:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.308 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5H0pP9yCjtoMl1W4TvUgcb8", 2020-11-29T16:11:12.309 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.340224_224.1", 2020-11-29T16:11:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.311 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-39", 2020-11-29T16:11:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.340224571Z", 2020-11-29T16:11:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Pp8cXl7djO1gxLhMq2AZXJp", 2020-11-29T16:11:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.314 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.343737_225.1", 2020-11-29T16:11:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-39", 2020-11-29T16:11:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.343737416Z", 2020-11-29T16:11:12.315 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Pp8cXl7djO1gxLhMq2AZXJp", 2020-11-29T16:11:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.368575_226.1", 2020-11-29T16:11:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-40:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40", 2020-11-29T16:11:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.368575114Z", 2020-11-29T16:11:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jgn-ZwFyVxv-tmED-Ggd4Hw", 2020-11-29T16:11:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.374510_227.1", 2020-11-29T16:11:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-40:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40", 2020-11-29T16:11:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.374510224Z", 2020-11-29T16:11:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jgn-ZwFyVxv-tmED-Ggd4Hw", 2020-11-29T16:11:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.376835_228.1", 2020-11-29T16:11:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-40", 2020-11-29T16:11:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.376835132Z", 2020-11-29T16:11:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qrn1nKXKg0aZIU2cqKpKgju", 2020-11-29T16:11:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.380733_229.1", 2020-11-29T16:11:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-40", 2020-11-29T16:11:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.380733954Z", 2020-11-29T16:11:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qrn1nKXKg0aZIU2cqKpKgju", 2020-11-29T16:11:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.333 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.430813_231.1", 2020-11-29T16:11:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-41:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41", 2020-11-29T16:11:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.430813301Z", 2020-11-29T16:11:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.334 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.335 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_upM-2-zq2ujNZABi5Lo2I6j", 2020-11-29T16:11:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.440999_232.1", 2020-11-29T16:11:12.337 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-41:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41", 2020-11-29T16:11:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.440999402Z", 2020-11-29T16:11:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.338 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_upM-2-zq2ujNZABi5Lo2I6j", 2020-11-29T16:11:12.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.341 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.444219_234.1", 2020-11-29T16:11:12.341 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.341 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-41", 2020-11-29T16:11:12.342 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.444219056Z", 2020-11-29T16:11:12.342 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.342 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.342 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.342 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.343 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.343 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iwxqSbv55pvq-WwQMf1oR3e", 2020-11-29T16:11:12.343 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.343 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.344 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:12.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:12.345 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.452875_235.1", 2020-11-29T16:11:12.345 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:12.345 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-41", 2020-11-29T16:11:12.345 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.452875466Z", 2020-11-29T16:11:12.345 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:12.346 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:12.346 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:12.346 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:12.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.346 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:12.347 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iwxqSbv55pvq-WwQMf1oR3e", 2020-11-29T16:11:12.347 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:12.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:12.347 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:12.347 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:12.348 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:12.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:12.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:12.350 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e24cddf9-396c-4261-a7bc-b4e1e678325e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:12.389 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:12.389 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:12.607 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:12.607 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.607 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665466.760427_46.1", 2020-11-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "user", 2020-11-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rgw-multisite-test-user", 2020-11-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:57:46.760427145Z", 2020-11-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.609 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.609 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.609 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.609 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.610 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7jrnGFazjfCMV9OgOp-5ICR", 2020-11-29T16:11:12.610 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.610 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.610 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.611 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.611 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.611 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.612 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.612 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.612 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665466.770600_48.1", 2020-11-29T16:11:12.612 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "user", 2020-11-29T16:11:12.612 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "rgw-multisite-test-user", 2020-11-29T16:11:12.613 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:57:46.770600943Z", 2020-11-29T16:11:12.613 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.613 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.613 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.614 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.614 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.614 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.614 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7jrnGFazjfCMV9OgOp-5ICR", 2020-11-29T16:11:12.614 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.615 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.615 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.615 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.615 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.615 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.616 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.616 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.616 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.309764_49.1", 2020-11-29T16:11:12.616 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.616 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-1:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1", 2020-11-29T16:11:12.617 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.309764005Z", 2020-11-29T16:11:12.617 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.617 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.617 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.618 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.618 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.618 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.618 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tDx7cRFyPylOsARNJShhjdi", 2020-11-29T16:11:12.618 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.619 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.619 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.619 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.619 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.619 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.620 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.620 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.620 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.329195_52.1", 2020-11-29T16:11:12.620 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.620 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-1:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1", 2020-11-29T16:11:12.621 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.329195289Z", 2020-11-29T16:11:12.621 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.622 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.622 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.622 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.623 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.623 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.623 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tDx7cRFyPylOsARNJShhjdi", 2020-11-29T16:11:12.624 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.624 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.625 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.625 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.625 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.626 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.626 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.627 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.627 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.330857_53.1", 2020-11-29T16:11:12.627 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.628 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-1", 2020-11-29T16:11:12.628 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.330857588Z", 2020-11-29T16:11:12.628 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.629 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.629 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.629 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.630 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.630 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.630 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_da31gfzBeEr_qTz8sD5Z95W", 2020-11-29T16:11:12.630 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.631 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.631 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.631 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.631 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.631 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.632 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.632 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.632 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.334069_54.1", 2020-11-29T16:11:12.632 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.633 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-1", 2020-11-29T16:11:12.633 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.334069131Z", 2020-11-29T16:11:12.633 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.633 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.633 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.634 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.634 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.634 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.634 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_da31gfzBeEr_qTz8sD5Z95W", 2020-11-29T16:11:12.634 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.635 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.635 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.635 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.635 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.635 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.636 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.636 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.636 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.361540_55.1", 2020-11-29T16:11:12.636 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.636 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-2:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2", 2020-11-29T16:11:12.637 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.361540407Z", 2020-11-29T16:11:12.637 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.637 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.637 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.638 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.638 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.638 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.638 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jcxJvuSk8NgQdLMU8LRCoaP", 2020-11-29T16:11:12.638 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.639 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.639 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.639 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.639 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.639 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.640 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.640 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.640 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.366215_56.1", 2020-11-29T16:11:12.640 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.640 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-2:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2", 2020-11-29T16:11:12.641 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.366215003Z", 2020-11-29T16:11:12.641 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.641 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.641 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.642 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.642 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.642 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.642 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jcxJvuSk8NgQdLMU8LRCoaP", 2020-11-29T16:11:12.642 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.643 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.643 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.643 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.643 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.643 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.644 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.644 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.644 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.369096_57.1", 2020-11-29T16:11:12.644 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.644 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-2", 2020-11-29T16:11:12.645 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.369096676Z", 2020-11-29T16:11:12.645 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.645 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.645 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.646 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.646 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.646 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.646 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_F0nqKIPOuZ20dNAraH9mMFO", 2020-11-29T16:11:12.646 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.647 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.647 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.647 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.647 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.648 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.648 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.649 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.649 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665590.374355_58.1", 2020-11-29T16:11:12.649 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.649 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-2", 2020-11-29T16:11:12.650 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:50.374355442Z", 2020-11-29T16:11:12.650 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.650 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.651 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.651 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.651 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.651 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_F0nqKIPOuZ20dNAraH9mMFO", 2020-11-29T16:11:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665593.914359_59.1", 2020-11-29T16:11:12.654 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.654 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-3:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3", 2020-11-29T16:11:12.654 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.914359282Z", 2020-11-29T16:11:12.654 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.655 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.655 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.655 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.656 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.656 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.656 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_nPQIea0SAu2YbHSwS1hqdTS", 2020-11-29T16:11:12.656 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.657 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.657 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.657 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.657 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.657 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.658 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.658 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.658 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665593.919600_60.1", 2020-11-29T16:11:12.659 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.659 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-3:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3", 2020-11-29T16:11:12.659 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.919600334Z", 2020-11-29T16:11:12.659 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.659 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.660 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.660 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.660 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.660 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.660 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_nPQIea0SAu2YbHSwS1hqdTS", 2020-11-29T16:11:12.661 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.661 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.661 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.661 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.661 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.662 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.662 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.662 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.663 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665593.924502_61.1", 2020-11-29T16:11:12.663 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.663 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-3", 2020-11-29T16:11:12.664 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.924502922Z", 2020-11-29T16:11:12.664 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.664 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.664 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.664 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.665 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.665 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.665 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_hgxnPf55gFvOJXZoH0wMw6D", 2020-11-29T16:11:12.665 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.665 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.666 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.666 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.666 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.666 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.667 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.667 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.667 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665593.929811_62.1", 2020-11-29T16:11:12.667 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.667 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-3", 2020-11-29T16:11:12.668 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:53.929811866Z", 2020-11-29T16:11:12.668 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.669 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.669 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.669 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.669 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.669 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.670 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_hgxnPf55gFvOJXZoH0wMw6D", 2020-11-29T16:11:12.670 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.670 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.670 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.671 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.671 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.671 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.671 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.671 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.672 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.255794_63.1", 2020-11-29T16:11:12.672 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.672 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-4:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4", 2020-11-29T16:11:12.672 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.255794146Z", 2020-11-29T16:11:12.672 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.673 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.673 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.673 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.673 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.673 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.674 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_dI-T3JKYYOksP3k9JHqk_4S", 2020-11-29T16:11:12.674 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.674 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.674 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.674 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.675 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.675 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.675 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.675 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.675 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.260821_65.1", 2020-11-29T16:11:12.676 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.676 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-4:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4", 2020-11-29T16:11:12.676 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.260821437Z", 2020-11-29T16:11:12.676 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.676 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.677 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.677 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.677 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.677 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.677 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_dI-T3JKYYOksP3k9JHqk_4S", 2020-11-29T16:11:12.678 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.678 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.678 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.678 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.679 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.679 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.679 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.679 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.679 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.262726_66.1", 2020-11-29T16:11:12.680 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.680 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-4", 2020-11-29T16:11:12.680 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.262726231Z", 2020-11-29T16:11:12.680 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.680 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.681 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.681 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.682 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.682 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.682 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_H0hQB2qGLgINO4xvzkf-CBV", 2020-11-29T16:11:12.683 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.683 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.683 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.683 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.683 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.684 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.684 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.684 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.684 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.267845_67.1", 2020-11-29T16:11:12.685 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.685 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-4", 2020-11-29T16:11:12.685 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.267845461Z", 2020-11-29T16:11:12.685 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.685 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.686 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.686 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.686 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.686 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.686 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_H0hQB2qGLgINO4xvzkf-CBV", 2020-11-29T16:11:12.687 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.687 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.687 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.687 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.687 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.688 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.688 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.688 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.688 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.291993_68.1", 2020-11-29T16:11:12.689 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.689 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-5:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5", 2020-11-29T16:11:12.690 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.291993232Z", 2020-11-29T16:11:12.690 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.690 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.691 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.691 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.691 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.691 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.691 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_eNmUNSw8O2Q1UqGf3-a8M0y", 2020-11-29T16:11:12.692 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.692 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.693 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.693 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.694 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.694 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.694 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.695 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.695 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.297549_69.1", 2020-11-29T16:11:12.695 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.695 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-5:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5", 2020-11-29T16:11:12.696 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.297549197Z", 2020-11-29T16:11:12.696 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.696 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.697 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.697 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.697 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.697 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.698 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_eNmUNSw8O2Q1UqGf3-a8M0y", 2020-11-29T16:11:12.698 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.698 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.698 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.699 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.699 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.699 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.700 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.700 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.700 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.299887_70.1", 2020-11-29T16:11:12.700 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.700 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-5", 2020-11-29T16:11:12.701 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.299887736Z", 2020-11-29T16:11:12.701 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.701 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.701 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.701 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.702 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.702 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.702 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_axHo0yOC2VpaRAyEFVHuReN", 2020-11-29T16:11:12.702 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.703 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.703 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.703 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.703 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.703 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.704 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.704 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.704 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.307514_71.1", 2020-11-29T16:11:12.704 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.704 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-5", 2020-11-29T16:11:12.705 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.307514021Z", 2020-11-29T16:11:12.705 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.705 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.706 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.706 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.707 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.707 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.707 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_axHo0yOC2VpaRAyEFVHuReN", 2020-11-29T16:11:12.707 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.707 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.708 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.708 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.708 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.708 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.709 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.709 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.709 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.352257_72.1", 2020-11-29T16:11:12.709 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.710 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-6:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6", 2020-11-29T16:11:12.710 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.352257512Z", 2020-11-29T16:11:12.710 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.710 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.710 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.711 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.711 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.711 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.711 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_nbxbaUi2Y7VYrWoa72raC5j", 2020-11-29T16:11:12.711 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.712 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.712 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.712 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.712 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.713 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.713 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.713 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.713 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.359127_73.1", 2020-11-29T16:11:12.713 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.714 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-6:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6", 2020-11-29T16:11:12.714 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.359127428Z", 2020-11-29T16:11:12.714 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.714 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.714 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.715 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.715 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.715 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.715 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_nbxbaUi2Y7VYrWoa72raC5j", 2020-11-29T16:11:12.716 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.716 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.717 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.717 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.717 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.718 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.718 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.718 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.718 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.361298_74.1", 2020-11-29T16:11:12.719 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.719 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-6", 2020-11-29T16:11:12.719 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.361298888Z", 2020-11-29T16:11:12.719 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.719 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.720 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.720 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.720 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.720 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.720 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5-Sqi-mEXlqXjQFGM13_N21", 2020-11-29T16:11:12.721 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.721 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.721 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.721 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.722 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.722 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.722 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.722 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.722 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665595.365399_75.1", 2020-11-29T16:11:12.723 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.723 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-6", 2020-11-29T16:11:12.723 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:55.365399707Z", 2020-11-29T16:11:12.723 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.724 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.724 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.724 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.724 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.724 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.725 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5-Sqi-mEXlqXjQFGM13_N21", 2020-11-29T16:11:12.725 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.725 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.725 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.725 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.726 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.726 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.726 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.726 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.726 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.250797_76.1", 2020-11-29T16:11:12.727 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.727 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7", 2020-11-29T16:11:12.727 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.250797689Z", 2020-11-29T16:11:12.727 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.727 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.728 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.728 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.728 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.728 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.728 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_T6kPv-3jveF29kZeCLJlo-6", 2020-11-29T16:11:12.729 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.729 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.729 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.729 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.730 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.730 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.730 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.730 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.730 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.255494_79.1", 2020-11-29T16:11:12.731 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.731 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7", 2020-11-29T16:11:12.731 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.255494488Z", 2020-11-29T16:11:12.732 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.732 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.732 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.733 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.733 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.733 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.733 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_T6kPv-3jveF29kZeCLJlo-6", 2020-11-29T16:11:12.733 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.734 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.734 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.734 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.734 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.734 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.735 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.735 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.735 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.257909_80.1", 2020-11-29T16:11:12.735 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.735 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7", 2020-11-29T16:11:12.736 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.257909992Z", 2020-11-29T16:11:12.736 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.736 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.736 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.737 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.737 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.737 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.737 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:12.738 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.738 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.738 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.738 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.738 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.739 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.739 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.739 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.739 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.265880_81.1", 2020-11-29T16:11:12.739 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.740 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7", 2020-11-29T16:11:12.740 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.265880801Z", 2020-11-29T16:11:12.740 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.740 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.741 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.741 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.741 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.741 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.742 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:12.742 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.742 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.742 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.742 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.743 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.743 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.743 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.743 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.743 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.284752_82.1", 2020-11-29T16:11:12.744 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.744 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8", 2020-11-29T16:11:12.744 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.284752329Z", 2020-11-29T16:11:12.744 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.745 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.745 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.745 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.745 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.745 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.746 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Qg8VBYmtvIuXfPsbR8FJmF5", 2020-11-29T16:11:12.746 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.746 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.746 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.746 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.747 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.747 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.747 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.747 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.747 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.290062_83.1", 2020-11-29T16:11:12.748 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.748 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8", 2020-11-29T16:11:12.748 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.290062502Z", 2020-11-29T16:11:12.748 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.748 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.749 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.749 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.749 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.749 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.749 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Qg8VBYmtvIuXfPsbR8FJmF5", 2020-11-29T16:11:12.750 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.750 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.750 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.750 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.751 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.751 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.751 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.751 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.751 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.292190_84.1", 2020-11-29T16:11:12.752 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.752 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8", 2020-11-29T16:11:12.752 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.292190927Z", 2020-11-29T16:11:12.752 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.752 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.753 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.753 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.753 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.753 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.754 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:12.754 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.754 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.754 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.754 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.755 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.755 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.755 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.755 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.756 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.296685_85.1", 2020-11-29T16:11:12.756 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.756 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8", 2020-11-29T16:11:12.756 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.296685079Z", 2020-11-29T16:11:12.756 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.757 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.757 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.757 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.758 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.758 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.758 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:12.759 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.759 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.759 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.759 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.759 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.760 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.760 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.760 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.760 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.338804_86.1", 2020-11-29T16:11:12.760 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.761 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9", 2020-11-29T16:11:12.761 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.338804685Z", 2020-11-29T16:11:12.761 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.761 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.761 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.762 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.762 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.762 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.762 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_x_puG7-2t3lKG0zWJ2e3lY4", 2020-11-29T16:11:12.763 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.763 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.763 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.763 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.763 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.764 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.764 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.764 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.764 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.345806_87.1", 2020-11-29T16:11:12.764 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.765 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9", 2020-11-29T16:11:12.765 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.345806856Z", 2020-11-29T16:11:12.765 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.765 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.765 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.766 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.766 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.766 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.766 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_x_puG7-2t3lKG0zWJ2e3lY4", 2020-11-29T16:11:12.767 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.767 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.767 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.768 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.768 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.768 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.768 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.769 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.769 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.348268_88.1", 2020-11-29T16:11:12.769 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.769 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9", 2020-11-29T16:11:12.769 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.348268690Z", 2020-11-29T16:11:12.770 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.770 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.770 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.770 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.770 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.771 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.771 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:12.771 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.771 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.772 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.772 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.772 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.773 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.773 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.773 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.774 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665598.354068_89.1", 2020-11-29T16:11:12.774 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.774 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9", 2020-11-29T16:11:12.774 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:58.354068281Z", 2020-11-29T16:11:12.774 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.775 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.775 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.775 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.775 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.776 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.776 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:12.776 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.776 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.776 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.777 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.777 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.777 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.777 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.777 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.778 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.682590_90.1", 2020-11-29T16:11:12.778 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.778 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7", 2020-11-29T16:11:12.778 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.682590411Z", 2020-11-29T16:11:12.778 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.779 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.779 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:12.779 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.779 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.779 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.780 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:12.780 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.780 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.780 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.780 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "remove" 2020-11-29T16:11:12.781 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.781 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.781 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.781 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.781 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.689390_92.1", 2020-11-29T16:11:12.782 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.782 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-7", 2020-11-29T16:11:12.782 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.689390648Z", 2020-11-29T16:11:12.782 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.783 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.783 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:12.783 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.783 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.783 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.784 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:12.784 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.784 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.784 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.788 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.788 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.788 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.789 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.789 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.789 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.764882_93.1", 2020-11-29T16:11:12.789 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.790 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8", 2020-11-29T16:11:12.790 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.764882801Z", 2020-11-29T16:11:12.790 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.790 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.790 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:12.791 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.791 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.791 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.791 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:12.791 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.792 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.792 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.792 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "remove" 2020-11-29T16:11:12.792 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.792 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.793 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.793 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.793 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.770654_94.1", 2020-11-29T16:11:12.793 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.794 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-8", 2020-11-29T16:11:12.794 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.770654401Z", 2020-11-29T16:11:12.794 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.794 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.794 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:12.795 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.795 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.795 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.795 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:12.795 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.796 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.796 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.796 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.796 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.797 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.797 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.797 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.798 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.889388_95.1", 2020-11-29T16:11:12.798 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.798 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9", 2020-11-29T16:11:12.798 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.889388291Z", 2020-11-29T16:11:12.799 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.799 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.799 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:12.800 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.800 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.800 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.801 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:12.801 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.801 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.801 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.802 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "remove" 2020-11-29T16:11:12.802 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.802 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.803 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.803 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.803 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665599.895400_96.1", 2020-11-29T16:11:12.803 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.803 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-9", 2020-11-29T16:11:12.804 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400726Z", 2020-11-29T16:11:12.804 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.804 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.804 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:12.805 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.805 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.805 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.805 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:12.805 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.806 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.806 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.806 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.806 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.807 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.807 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.807 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.807 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.191369_34.1", 2020-11-29T16:11:12.808 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.808 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10", 2020-11-29T16:11:12.808 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.191369861Z", 2020-11-29T16:11:12.808 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.809 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.809 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.809 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.810 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.810 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.810 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YrVXr7ERmibtK1P81HX79vZ", 2020-11-29T16:11:12.810 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.810 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.811 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.811 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.811 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.811 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.195873_35.1", 2020-11-29T16:11:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10", 2020-11-29T16:11:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.195873130Z", 2020-11-29T16:11:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.814 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.814 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.814 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.814 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YrVXr7ERmibtK1P81HX79vZ", 2020-11-29T16:11:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.198497_36.1", 2020-11-29T16:11:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-10", 2020-11-29T16:11:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.198497073Z", 2020-11-29T16:11:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_DFiygE422CgLw5QBztkfzu-", 2020-11-29T16:11:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.204454_37.1", 2020-11-29T16:11:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-10", 2020-11-29T16:11:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.204454835Z", 2020-11-29T16:11:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_DFiygE422CgLw5QBztkfzu-", 2020-11-29T16:11:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.825 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.229632_38.1", 2020-11-29T16:11:12.825 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.825 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11", 2020-11-29T16:11:12.825 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.229632159Z", 2020-11-29T16:11:12.825 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.826 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.826 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.826 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.826 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.826 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.827 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_20WS3iYLQJHPayxR2Iu8xxa", 2020-11-29T16:11:12.827 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.827 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.827 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.827 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.828 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.828 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.828 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.828 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.828 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.234831_39.1", 2020-11-29T16:11:12.829 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.829 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11", 2020-11-29T16:11:12.829 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.234831409Z", 2020-11-29T16:11:12.829 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.830 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.830 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.830 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.830 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.830 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.831 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_20WS3iYLQJHPayxR2Iu8xxa", 2020-11-29T16:11:12.831 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.831 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.831 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.831 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.832 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.832 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.832 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.832 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.832 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.237152_40.1", 2020-11-29T16:11:12.833 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.833 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-11", 2020-11-29T16:11:12.833 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.237152720Z", 2020-11-29T16:11:12.833 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.833 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.834 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.834 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.834 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.834 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.835 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_0w2vqVYxXcNZ-dU0Fiwn2qb", 2020-11-29T16:11:12.835 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.835 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.835 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.836 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.836 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.836 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.836 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.837 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.837 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.241905_41.1", 2020-11-29T16:11:12.837 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.837 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-11", 2020-11-29T16:11:12.837 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.241905299Z", 2020-11-29T16:11:12.838 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.838 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.838 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.838 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.838 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.839 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.839 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_0w2vqVYxXcNZ-dU0Fiwn2qb", 2020-11-29T16:11:12.839 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.839 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.840 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.840 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.840 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.840 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.840 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.841 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.841 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.311228_42.1", 2020-11-29T16:11:12.841 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.841 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12", 2020-11-29T16:11:12.842 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.311228153Z", 2020-11-29T16:11:12.842 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.842 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.842 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.842 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.843 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.843 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.843 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jrQUaYBzA_aLdZ7o_IQP6PE", 2020-11-29T16:11:12.843 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.843 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.844 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.844 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.844 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.844 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.844 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.845 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.845 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.320192_43.1", 2020-11-29T16:11:12.845 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.845 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12", 2020-11-29T16:11:12.846 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.320192323Z", 2020-11-29T16:11:12.846 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.846 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.846 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.846 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.847 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.847 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.847 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jrQUaYBzA_aLdZ7o_IQP6PE", 2020-11-29T16:11:12.847 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.847 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.848 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.848 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.848 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.848 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.848 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.849 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.849 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.328671_44.1", 2020-11-29T16:11:12.849 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.849 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-12", 2020-11-29T16:11:12.849 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.328671604Z", 2020-11-29T16:11:12.850 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.850 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.850 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.850 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.850 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.851 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.851 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5MnB5hm2ISAIj9JGdxV50H9", 2020-11-29T16:11:12.851 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.851 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.852 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.852 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.852 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.852 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.852 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.853 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.853 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665601.335813_45.1", 2020-11-29T16:11:12.853 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.853 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-12", 2020-11-29T16:11:12.853 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:01.335813349Z", 2020-11-29T16:11:12.854 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.854 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.854 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.854 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.854 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.855 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.855 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5MnB5hm2ISAIj9JGdxV50H9", 2020-11-29T16:11:12.855 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.855 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.856 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.856 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.856 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.856 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.857 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.857 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.857 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.841649_46.1", 2020-11-29T16:11:12.857 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.857 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13", 2020-11-29T16:11:12.858 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.841649100Z", 2020-11-29T16:11:12.858 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.858 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.858 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.858 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.859 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.859 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.859 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Kqyzq54nxbVSdNcuPFIuxtb", 2020-11-29T16:11:12.859 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.859 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.860 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.860 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.860 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.860 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.860 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.861 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.861 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.847258_47.1", 2020-11-29T16:11:12.861 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.861 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13", 2020-11-29T16:11:12.861 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.847258533Z", 2020-11-29T16:11:12.862 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.862 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.862 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.862 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.863 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.863 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.863 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Kqyzq54nxbVSdNcuPFIuxtb", 2020-11-29T16:11:12.863 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.863 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.864 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.864 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.864 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.864 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.864 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.865 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.865 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.849488_48.1", 2020-11-29T16:11:12.865 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.865 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-13", 2020-11-29T16:11:12.865 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.849488010Z", 2020-11-29T16:11:12.866 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.866 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.866 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.866 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.867 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.867 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.867 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rjVcqahbW1Rw3RINe760YzT", 2020-11-29T16:11:12.867 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.868 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.868 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.868 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.869 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.869 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.869 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.869 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.869 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.853552_49.1", 2020-11-29T16:11:12.870 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.870 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-13", 2020-11-29T16:11:12.870 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.853552156Z", 2020-11-29T16:11:12.870 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.870 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.871 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.871 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.871 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.871 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.872 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rjVcqahbW1Rw3RINe760YzT", 2020-11-29T16:11:12.872 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.872 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.872 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.873 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.873 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.873 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.873 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.874 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.874 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.876908_50.1", 2020-11-29T16:11:12.874 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.875 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14", 2020-11-29T16:11:12.875 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.876908996Z", 2020-11-29T16:11:12.875 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.875 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.875 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.876 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.876 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.876 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.876 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YVkRWUTasfPQtpHA7NklgCE", 2020-11-29T16:11:12.876 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.877 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.877 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.877 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.877 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.877 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.878 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.878 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.878 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.881320_51.1", 2020-11-29T16:11:12.878 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.879 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14", 2020-11-29T16:11:12.879 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.881320871Z", 2020-11-29T16:11:12.879 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.879 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.879 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.880 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.880 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.880 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.880 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YVkRWUTasfPQtpHA7NklgCE", 2020-11-29T16:11:12.880 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.881 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.881 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.881 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.881 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.881 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.882 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.882 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.882 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.883170_52.1", 2020-11-29T16:11:12.882 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.882 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-14", 2020-11-29T16:11:12.883 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.883170592Z", 2020-11-29T16:11:12.883 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.883 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.883 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.884 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.884 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.884 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.884 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AsuYPfubiijDBKRDJK6qqjt", 2020-11-29T16:11:12.884 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.885 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.885 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.885 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.885 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.885 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.886 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.886 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.886 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.887114_53.1", 2020-11-29T16:11:12.886 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.886 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-14", 2020-11-29T16:11:12.887 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.887114146Z", 2020-11-29T16:11:12.887 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.887 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.887 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.888 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.888 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.888 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.888 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AsuYPfubiijDBKRDJK6qqjt", 2020-11-29T16:11:12.889 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.889 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.889 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.889 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.889 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.890 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.890 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.890 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.890 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.932301_54.1", 2020-11-29T16:11:12.890 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.891 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15", 2020-11-29T16:11:12.891 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.932301379Z", 2020-11-29T16:11:12.891 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.891 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.891 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.892 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.892 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.892 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.892 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_x_VdKNwXX4kJXxzVcmjduF6", 2020-11-29T16:11:12.892 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.893 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.893 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.893 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.893 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.893 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.894 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.894 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.894 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.936160_55.1", 2020-11-29T16:11:12.894 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.895 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15", 2020-11-29T16:11:12.895 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.936160887Z", 2020-11-29T16:11:12.895 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.895 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.895 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.896 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.896 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.896 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.896 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_x_VdKNwXX4kJXxzVcmjduF6", 2020-11-29T16:11:12.896 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.897 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.897 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.897 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.897 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.897 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.898 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.898 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.898 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.937891_56.1", 2020-11-29T16:11:12.898 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.898 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-15", 2020-11-29T16:11:12.899 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.937891458Z", 2020-11-29T16:11:12.899 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.899 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.899 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.900 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.900 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.900 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.900 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_QoRko9LJwKUXdid0giAhASQ", 2020-11-29T16:11:12.900 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.901 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.901 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.901 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.901 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.901 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.902 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.902 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.902 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665612.941872_57.1", 2020-11-29T16:11:12.902 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.903 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-15", 2020-11-29T16:11:12.903 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:00:12.941872760Z", 2020-11-29T16:11:12.903 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.903 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.903 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.904 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.904 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.904 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.904 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_QoRko9LJwKUXdid0giAhASQ", 2020-11-29T16:11:12.904 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.905 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.905 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.905 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.905 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.905 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.906 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.906 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.906 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.057028_59.1", 2020-11-29T16:11:12.906 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.907 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:12.907 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.057028183Z", 2020-11-29T16:11:12.907 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.907 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.908 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.908 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.908 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.908 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.909 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:12.909 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.909 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.909 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.910 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.910 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.910 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.910 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.911 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.911 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.061373_60.1", 2020-11-29T16:11:12.911 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.911 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:12.911 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.061373004Z", 2020-11-29T16:11:12.912 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.912 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.912 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.912 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.913 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.913 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.913 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:12.913 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.914 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.914 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.914 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.914 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.915 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.915 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.915 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.915 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.063911_61.1", 2020-11-29T16:11:12.915 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.916 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16", 2020-11-29T16:11:12.916 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.063911936Z", 2020-11-29T16:11:12.916 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.916 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.916 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.917 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.917 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.917 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.917 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_qWQvPnGcRTlfP8YPDDWPU7j", 2020-11-29T16:11:12.918 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.918 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.918 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.918 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.918 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.919 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.919 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.919 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.919 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.069446_62.1", 2020-11-29T16:11:12.919 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.920 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16", 2020-11-29T16:11:12.920 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.069446267Z", 2020-11-29T16:11:12.920 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.920 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.921 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.921 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.921 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.921 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.921 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_qWQvPnGcRTlfP8YPDDWPU7j", 2020-11-29T16:11:12.922 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.922 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.922 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.922 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.923 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.923 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.923 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.923 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.923 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.097339_63.1", 2020-11-29T16:11:12.924 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.924 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:12.924 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.097339684Z", 2020-11-29T16:11:12.924 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.925 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.925 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.925 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.925 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.925 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.926 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:12.926 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.926 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.926 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.926 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.927 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.927 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.927 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.927 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.927 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.102188_64.1", 2020-11-29T16:11:12.928 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.928 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:12.928 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.102188782Z", 2020-11-29T16:11:12.928 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.928 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.929 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.929 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.929 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.929 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.929 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:12.930 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.930 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.930 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.930 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.931 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.931 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.931 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.931 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.931 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.104862_65.1", 2020-11-29T16:11:12.932 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.932 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17", 2020-11-29T16:11:12.932 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.104862905Z", 2020-11-29T16:11:12.932 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.932 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.933 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.933 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.933 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.933 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.933 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3YDKF2LZHEWfgfUO2EvjsgL", 2020-11-29T16:11:12.934 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.934 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.934 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.934 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.935 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.935 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.935 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.935 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.935 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.108626_66.1", 2020-11-29T16:11:12.936 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.936 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17", 2020-11-29T16:11:12.936 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.108626431Z", 2020-11-29T16:11:12.936 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.937 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.937 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.937 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.938 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.938 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.938 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3YDKF2LZHEWfgfUO2EvjsgL", 2020-11-29T16:11:12.938 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.939 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.939 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.939 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.939 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.939 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.940 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.940 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.940 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.150629_67.1", 2020-11-29T16:11:12.940 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.941 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:12.941 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.150629298Z", 2020-11-29T16:11:12.941 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.941 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.942 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.942 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.942 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.942 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.942 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:12.943 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.943 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.943 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.943 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.944 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.944 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.944 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.944 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.944 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.158738_68.1", 2020-11-29T16:11:12.945 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.945 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:12.945 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.158738425Z", 2020-11-29T16:11:12.945 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.945 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.946 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.946 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.946 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.946 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.947 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:12.947 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.947 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.947 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.947 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.948 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.948 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.948 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.948 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.948 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.160845_69.1", 2020-11-29T16:11:12.949 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.949 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18", 2020-11-29T16:11:12.949 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.160845058Z", 2020-11-29T16:11:12.949 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.949 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.950 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.950 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.950 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.950 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.951 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5xSDI0jeDBdQUr4kb2mMHId", 2020-11-29T16:11:12.951 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.951 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.951 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.951 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.952 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.952 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.952 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.952 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.952 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.164732_70.1", 2020-11-29T16:11:12.953 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.953 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18", 2020-11-29T16:11:12.953 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.164732376Z", 2020-11-29T16:11:12.953 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.953 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.954 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.954 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.954 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.954 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.954 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5xSDI0jeDBdQUr4kb2mMHId", 2020-11-29T16:11:12.955 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.955 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.955 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.955 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.956 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.956 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.956 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.956 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.957 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.187080_71.1", 2020-11-29T16:11:12.957 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.957 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:12.957 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.187080319Z", 2020-11-29T16:11:12.957 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.958 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.958 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:12.958 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.958 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.958 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.959 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:12.959 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.959 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.959 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.960 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.960 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.960 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.960 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.960 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.961 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.195335_72.1", 2020-11-29T16:11:12.961 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.961 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:12.961 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.195335470Z", 2020-11-29T16:11:12.961 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.962 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.962 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:12.962 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.962 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.962 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.963 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:12.963 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.963 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.963 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.963 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.964 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.964 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.964 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.964 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.964 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.203159_73.1", 2020-11-29T16:11:12.965 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.965 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:12.965 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.203159941Z", 2020-11-29T16:11:12.965 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.966 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.966 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:12.966 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.966 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.966 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.967 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:12.967 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.967 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.967 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.967 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.968 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.968 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.968 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.968 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.969 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.210320_74.1", 2020-11-29T16:11:12.969 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.969 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:12.969 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.210320129Z", 2020-11-29T16:11:12.970 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.970 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.970 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:12.970 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.970 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.971 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.971 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:12.971 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.971 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.972 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.972 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.972 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.972 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.972 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.973 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.973 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.221591_75.1", 2020-11-29T16:11:12.973 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.973 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:12.974 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.221591576Z", 2020-11-29T16:11:12.974 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.974 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.975 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:12.975 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.975 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.975 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.976 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:12.976 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.976 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.976 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.976 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.977 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.977 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.977 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.977 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.977 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665662.229493_76.1", 2020-11-29T16:11:12.978 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.978 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:12.978 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:01:02.229493483Z", 2020-11-29T16:11:12.978 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.978 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.979 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:12.979 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.979 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.979 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.979 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:12.980 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:12.980 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.980 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.980 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.981 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.981 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.981 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.981 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.981 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.967652_86.1", 2020-11-29T16:11:12.982 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.982 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:12.982 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.967652642Z", 2020-11-29T16:11:12.982 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.982 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.983 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.983 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.983 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.983 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.983 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:12.984 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.984 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.984 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.984 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.984 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.985 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.985 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.985 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.985 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.972279_87.1", 2020-11-29T16:11:12.986 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.986 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:12.986 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.972279097Z", 2020-11-29T16:11:12.986 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.986 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.987 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.987 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.987 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.987 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.987 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:12.988 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.988 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.989 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.989 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.989 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.990 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.990 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.990 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.990 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.974314_88.1", 2020-11-29T16:11:12.991 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.991 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19", 2020-11-29T16:11:12.991 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.974314737Z", 2020-11-29T16:11:12.991 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.992 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.992 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.992 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.992 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.992 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.993 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_iGyiAtOMEjJAj1PjCXsPDZz", 2020-11-29T16:11:12.993 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.993 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.993 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.993 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:12.994 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.994 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.994 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.994 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.994 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.978564_89.1", 2020-11-29T16:11:12.995 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:12.995 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19", 2020-11-29T16:11:12.995 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.978564499Z", 2020-11-29T16:11:12.995 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.995 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:12.996 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:12.996 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:12.996 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.996 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:12.997 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_iGyiAtOMEjJAj1PjCXsPDZz", 2020-11-29T16:11:12.997 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:12.997 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.997 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:12.997 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:12.998 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.998 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:12.998 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:12.998 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:12.998 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.984258_90.1", 2020-11-29T16:11:12.999 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:12.999 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:12.999 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.984258860Z", 2020-11-29T16:11:12.999 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:12.999 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.000 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:13.000 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.000 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.000 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.000 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:13.001 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.001 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.001 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.001 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.001 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.002 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.002 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.002 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.002 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665856.991097_91.1", 2020-11-29T16:11:13.002 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.003 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:13.003 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:16.991097909Z", 2020-11-29T16:11:13.003 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.003 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.004 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:13.004 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.004 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.004 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.004 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:13.005 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.005 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.005 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.005 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.005 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.006 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.006 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.006 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.006 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665997.215638_94.1", 2020-11-29T16:11:13.006 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.007 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:13.007 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.215638352Z", 2020-11-29T16:11:13.007 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.007 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.008 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.008 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.008 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.008 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.009 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.009 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.009 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.009 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.010 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.010 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.010 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.010 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.011 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.011 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665997.220413_95.1", 2020-11-29T16:11:13.011 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.011 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:13.012 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.220413426Z", 2020-11-29T16:11:13.012 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.012 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.012 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.012 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.013 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.013 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.013 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.013 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.013 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.014 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.014 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.014 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.014 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.014 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.015 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.015 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665997.222814_96.1", 2020-11-29T16:11:13.015 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.015 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20", 2020-11-29T16:11:13.015 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.222814425Z", 2020-11-29T16:11:13.016 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.016 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.016 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.016 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.017 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.017 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.017 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_caPUfpnBPovw7pnaLhfQKQu", 2020-11-29T16:11:13.017 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.017 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.018 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.018 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.018 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.018 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.018 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.019 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.019 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606665997.230793_97.1", 2020-11-29T16:11:13.019 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.019 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20", 2020-11-29T16:11:13.019 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:37.230793956Z", 2020-11-29T16:11:13.020 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.020 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.020 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.020 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.020 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.021 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.021 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_caPUfpnBPovw7pnaLhfQKQu", 2020-11-29T16:11:13.021 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.021 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.021 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.022 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.022 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.022 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.022 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.023 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.023 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666001.819331_99.1", 2020-11-29T16:11:13.023 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.023 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:13.024 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:41.819331435Z", 2020-11-29T16:11:13.024 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.024 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.024 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.024 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.025 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.025 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.025 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.025 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.025 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.026 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.026 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.026 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.026 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.026 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.027 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.027 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666001.830749_100.1", 2020-11-29T16:11:13.027 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.027 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:13.028 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:06:41.830749663Z", 2020-11-29T16:11:13.028 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.028 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.028 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.028 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.029 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.029 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.029 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.029 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.029 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.030 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.030 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.030 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.030 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.030 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.031 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.031 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666061.809169_106.1", 2020-11-29T16:11:13.031 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.031 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:13.031 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:07:41.809169955Z", 2020-11-29T16:11:13.032 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.032 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.032 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.032 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.033 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.033 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.033 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.033 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:13.033 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.034 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.034 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.034 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.034 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.034 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.035 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.035 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666061.819824_107.1", 2020-11-29T16:11:13.035 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.035 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:13.035 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:07:41.819824059Z", 2020-11-29T16:11:13.036 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.036 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.036 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.036 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.036 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.037 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.037 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:13.037 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:13.037 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.037 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.038 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.038 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.038 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.038 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.038 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.039 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.732730_108.1", 2020-11-29T16:11:13.039 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.039 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:13.039 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.732730870Z", 2020-11-29T16:11:13.039 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.040 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.040 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.040 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.040 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.040 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.041 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:13.041 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.041 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.041 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.042 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.042 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.042 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.042 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.042 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.043 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.738936_109.1", 2020-11-29T16:11:13.043 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.043 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:13.043 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.738936839Z", 2020-11-29T16:11:13.043 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.044 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.044 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.044 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.044 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.044 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.045 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:13.045 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.045 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.045 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.045 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.046 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.046 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.046 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.046 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.047 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.740862_110.1", 2020-11-29T16:11:13.047 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.047 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21", 2020-11-29T16:11:13.047 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.740862527Z", 2020-11-29T16:11:13.047 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.048 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.048 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.049 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.049 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.049 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.050 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1LtU-sUHWZQfFCaHHia4XL6", 2020-11-29T16:11:13.050 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.050 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.051 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.051 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.051 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.051 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.052 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.052 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.052 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.745932_111.1", 2020-11-29T16:11:13.052 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.052 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21", 2020-11-29T16:11:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.745932979Z", 2020-11-29T16:11:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.054 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.054 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.054 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1LtU-sUHWZQfFCaHHia4XL6", 2020-11-29T16:11:13.054 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.056 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.056 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.056 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.771777_112.1", 2020-11-29T16:11:13.056 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.057 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:13.057 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.771777245Z", 2020-11-29T16:11:13.057 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.057 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.057 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.058 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.058 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.058 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.058 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:13.059 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.059 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.059 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.059 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.059 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.060 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.060 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.060 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.060 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.779214_113.1", 2020-11-29T16:11:13.061 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.061 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:13.062 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.779214800Z", 2020-11-29T16:11:13.062 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.062 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.063 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.063 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.063 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.064 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.064 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:13.065 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.065 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.065 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.066 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.066 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.066 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.067 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.067 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.067 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.781033_114.1", 2020-11-29T16:11:13.068 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.068 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22", 2020-11-29T16:11:13.068 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.781033305Z", 2020-11-29T16:11:13.068 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.069 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.069 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.069 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.070 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.070 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.070 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_yAIfpIe3NidR6qhQ52Y6gQ9", 2020-11-29T16:11:13.071 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.071 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.071 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.071 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.072 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.072 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.072 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.072 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.072 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.786435_115.1", 2020-11-29T16:11:13.073 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.073 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22", 2020-11-29T16:11:13.073 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.786435846Z", 2020-11-29T16:11:13.073 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.074 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.074 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.074 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.074 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.075 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.075 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_yAIfpIe3NidR6qhQ52Y6gQ9", 2020-11-29T16:11:13.075 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.075 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.076 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.076 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.076 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.076 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.076 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.077 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.077 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.833583_116.1", 2020-11-29T16:11:13.077 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.077 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:13.078 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.833583970Z", 2020-11-29T16:11:13.078 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.078 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.078 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.078 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.079 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.079 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.079 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:13.079 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.079 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.080 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.080 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.080 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.080 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.080 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.081 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.081 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.838249_117.1", 2020-11-29T16:11:13.081 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.081 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:13.082 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.838249602Z", 2020-11-29T16:11:13.082 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.082 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.082 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.082 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.083 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.083 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.083 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:13.083 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.083 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.084 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.084 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.084 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.084 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.084 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.085 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.085 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.840897_118.1", 2020-11-29T16:11:13.085 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.085 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23", 2020-11-29T16:11:13.086 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.840897485Z", 2020-11-29T16:11:13.086 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.086 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.086 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.086 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.087 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.087 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.087 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_yUDBs52ibfejiizLQQANf-x", 2020-11-29T16:11:13.087 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.088 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.088 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.088 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.088 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.088 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.089 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.089 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.089 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.846339_119.1", 2020-11-29T16:11:13.089 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.090 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23", 2020-11-29T16:11:13.090 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.846339269Z", 2020-11-29T16:11:13.090 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.090 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.090 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.091 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.091 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.091 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.091 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_yUDBs52ibfejiizLQQANf-x", 2020-11-29T16:11:13.091 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.092 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.092 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.092 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.092 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.093 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.093 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.093 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.093 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.869230_120.1", 2020-11-29T16:11:13.093 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.094 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:13.094 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.869230022Z", 2020-11-29T16:11:13.094 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.094 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.094 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:13.095 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.095 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.095 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.095 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:13.095 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.096 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.096 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.096 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.096 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.097 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.097 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.097 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.097 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.875173_121.1", 2020-11-29T16:11:13.097 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.098 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:13.098 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.875173699Z", 2020-11-29T16:11:13.098 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.098 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.099 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:13.099 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.099 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.099 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.099 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:13.100 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.100 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.100 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.100 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.101 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.101 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.101 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.101 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.101 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.879784_122.1", 2020-11-29T16:11:13.102 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.102 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:13.102 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.879784537Z", 2020-11-29T16:11:13.102 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.102 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.103 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:13.103 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.103 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.103 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.103 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:13.104 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.104 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.104 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.104 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.104 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.105 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.105 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.105 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.105 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.886953_123.1", 2020-11-29T16:11:13.106 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.106 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:13.106 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.886953932Z", 2020-11-29T16:11:13.106 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.106 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.107 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:13.107 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.107 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.107 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.107 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:13.108 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.108 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.108 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.109 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.109 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.109 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.109 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.109 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.110 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.901158_124.1", 2020-11-29T16:11:13.110 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.110 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:13.110 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.901158656Z", 2020-11-29T16:11:13.111 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.111 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.111 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:13.111 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.112 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.112 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.112 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:13.113 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.113 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.114 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.114 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.114 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.114 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.115 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.115 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.115 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666101.915755_125.1", 2020-11-29T16:11:13.115 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.116 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:13.116 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:21.915755723Z", 2020-11-29T16:11:13.116 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.116 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.117 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:13.117 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.117 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.117 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.117 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:13.118 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.118 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.118 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.118 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.119 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.119 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.119 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.119 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.119 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.456443_126.1", 2020-11-29T16:11:13.120 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.120 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:13.120 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.456443611Z", 2020-11-29T16:11:13.120 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.120 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.121 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.121 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.121 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.121 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.121 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.122 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.122 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.122 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.122 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.122 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.123 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.123 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.123 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.123 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.460387_127.1", 2020-11-29T16:11:13.124 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.124 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:13.124 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.460387996Z", 2020-11-29T16:11:13.124 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.124 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.125 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.125 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.125 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.125 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.125 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.126 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.126 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.126 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.126 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.126 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.127 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.127 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.127 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.127 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.462193_128.1", 2020-11-29T16:11:13.127 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.128 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24", 2020-11-29T16:11:13.128 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.462193185Z", 2020-11-29T16:11:13.128 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.128 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.129 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.129 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.129 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.129 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.129 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_8rFe9Ex-HuV_dw_B2ecHBhB", 2020-11-29T16:11:13.130 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.130 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.130 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.130 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.130 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.131 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.131 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.131 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.131 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.467714_129.1", 2020-11-29T16:11:13.131 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.132 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24", 2020-11-29T16:11:13.132 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.467714061Z", 2020-11-29T16:11:13.132 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.132 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.132 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.133 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.133 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.133 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.133 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_8rFe9Ex-HuV_dw_B2ecHBhB", 2020-11-29T16:11:13.134 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.134 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.134 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.134 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.134 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.135 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.135 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.135 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.135 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.490669_130.1", 2020-11-29T16:11:13.135 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.136 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:13.136 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.490669098Z", 2020-11-29T16:11:13.136 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.136 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.136 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.137 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.137 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.137 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.137 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.137 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.138 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.138 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.138 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.138 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.139 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.139 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.139 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.139 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.496194_131.1", 2020-11-29T16:11:13.139 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.140 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:13.140 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.496194848Z", 2020-11-29T16:11:13.140 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.140 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.140 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.141 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.141 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.141 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.141 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.141 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.142 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.142 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.142 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.142 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.142 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.143 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.143 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.143 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.498339_132.1", 2020-11-29T16:11:13.143 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.144 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25", 2020-11-29T16:11:13.144 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.498339597Z", 2020-11-29T16:11:13.144 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.144 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.144 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.145 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.145 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.145 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.145 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_s63DKGRwg1GnITE9VweheW4", 2020-11-29T16:11:13.145 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.146 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.146 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.146 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.146 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.146 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.147 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.147 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.147 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.502532_133.1", 2020-11-29T16:11:13.147 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.147 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25", 2020-11-29T16:11:13.148 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.502532461Z", 2020-11-29T16:11:13.148 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.148 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.148 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.148 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.149 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.149 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.149 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_s63DKGRwg1GnITE9VweheW4", 2020-11-29T16:11:13.149 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.149 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.150 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.150 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.150 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.150 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.151 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.151 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.151 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.557238_134.1", 2020-11-29T16:11:13.151 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.151 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:13.152 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.557238717Z", 2020-11-29T16:11:13.152 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.152 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.152 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.152 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.153 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.153 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.153 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.153 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.153 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.154 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.154 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.154 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.154 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.154 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.155 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.155 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.562930_135.1", 2020-11-29T16:11:13.155 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.155 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:13.156 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.562930060Z", 2020-11-29T16:11:13.156 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.156 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.156 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.156 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.157 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.157 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.157 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.157 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.158 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.158 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.158 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.158 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.158 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.159 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.159 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.159 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.565930_136.1", 2020-11-29T16:11:13.159 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.159 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26", 2020-11-29T16:11:13.160 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.565930147Z", 2020-11-29T16:11:13.160 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.160 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.160 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.161 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.161 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.161 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.161 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_HEKlnFCl4gKqxl9FXC6_Slr", 2020-11-29T16:11:13.161 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.162 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.162 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.162 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.162 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.163 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.163 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.163 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.163 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.574718_137.1", 2020-11-29T16:11:13.164 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.164 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26", 2020-11-29T16:11:13.164 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.574718238Z", 2020-11-29T16:11:13.164 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.165 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.165 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.165 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.165 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.165 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.166 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_HEKlnFCl4gKqxl9FXC6_Slr", 2020-11-29T16:11:13.166 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.166 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.166 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.166 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.167 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.167 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.167 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.167 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.167 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.594772_138.1", 2020-11-29T16:11:13.168 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.168 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:13.168 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.594772332Z", 2020-11-29T16:11:13.168 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.169 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.169 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.169 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.169 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.169 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.170 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.170 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.170 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.170 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.170 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.171 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.171 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.171 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.172 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.172 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.599622_139.1", 2020-11-29T16:11:13.172 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.172 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:13.172 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.599622375Z", 2020-11-29T16:11:13.173 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.173 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.173 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.173 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.173 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.174 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.174 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.174 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.175 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.175 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.176 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.176 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.176 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.177 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.177 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.178 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.603089_140.1", 2020-11-29T16:11:13.178 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.178 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:13.179 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.603089353Z", 2020-11-29T16:11:13.179 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.180 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.180 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.180 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.181 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.181 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.182 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.182 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:13.182 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.183 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.183 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.184 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.184 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.184 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.185 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.185 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.612024_141.1", 2020-11-29T16:11:13.186 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.186 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:13.187 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.612024636Z", 2020-11-29T16:11:13.187 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.187 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.188 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.188 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.189 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.189 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.189 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:13.190 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:13.190 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.191 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.191 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.191 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.192 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.192 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.193 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.193 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.616582_142.1", 2020-11-29T16:11:13.194 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.194 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:13.194 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.616582099Z", 2020-11-29T16:11:13.195 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.195 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.196 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.196 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.196 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.197 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.197 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.198 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.198 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.199 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.199 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.199 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.200 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.200 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.201 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.201 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.624065_143.1", 2020-11-29T16:11:13.201 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.202 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:13.202 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.624065742Z", 2020-11-29T16:11:13.203 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.203 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.204 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.204 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.204 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.205 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.205 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.206 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.206 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.206 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.207 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.207 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.208 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.208 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.208 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.209 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.633769_144.1", 2020-11-29T16:11:13.209 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.210 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:13.210 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.633769558Z", 2020-11-29T16:11:13.211 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.211 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.211 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.212 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.212 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.213 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.213 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.214 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:13.214 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.214 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.215 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.215 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.216 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.216 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.216 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.217 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.641875_145.1", 2020-11-29T16:11:13.217 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.217 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:13.218 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.641875589Z", 2020-11-29T16:11:13.218 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.219 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.219 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.219 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.220 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.220 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.220 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:13.220 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:13.221 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.221 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.221 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.221 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.221 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.222 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.222 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.222 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.655938_146.1", 2020-11-29T16:11:13.222 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.222 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:13.223 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.655938546Z", 2020-11-29T16:11:13.223 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.223 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.223 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.223 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.224 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.224 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.224 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.224 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.224 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.225 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.225 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.225 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.225 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.225 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.226 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.226 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.665066_147.1", 2020-11-29T16:11:13.226 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.226 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:13.226 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.665066109Z", 2020-11-29T16:11:13.226 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.227 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.227 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.227 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.227 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.227 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.228 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.228 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.228 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.228 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.228 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.229 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.229 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.229 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.229 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.229 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.677379_148.1", 2020-11-29T16:11:13.229 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.230 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:13.230 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.677379068Z", 2020-11-29T16:11:13.230 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.230 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.230 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.231 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.231 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.231 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.231 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.231 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:13.232 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.232 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.232 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.232 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.232 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.233 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.233 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.233 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666112.684234_149.1", 2020-11-29T16:11:13.233 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.233 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:13.233 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:32.684234315Z", 2020-11-29T16:11:13.234 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.234 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.234 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.234 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.234 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.235 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.235 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:13.235 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 3 2020-11-29T16:11:13.235 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.235 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.236 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.236 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.236 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.236 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.236 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.237 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.338993_154.1", 2020-11-29T16:11:13.237 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.237 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:13.237 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.338993057Z", 2020-11-29T16:11:13.237 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.238 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.238 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.238 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.238 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.238 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.238 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:13.239 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.239 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.239 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.239 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.239 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.240 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.240 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.240 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.240 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.346126_155.1", 2020-11-29T16:11:13.240 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.241 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:13.241 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.346126383Z", 2020-11-29T16:11:13.241 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.241 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.241 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.242 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.242 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.242 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.242 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:13.242 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.243 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.243 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.243 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.243 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.243 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.243 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.244 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.244 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.348222_156.1", 2020-11-29T16:11:13.244 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.244 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27", 2020-11-29T16:11:13.244 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.348222396Z", 2020-11-29T16:11:13.245 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.245 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.245 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.245 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.245 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.246 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.246 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Wc4auwc30IepX657G2eZM-u", 2020-11-29T16:11:13.246 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.246 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.246 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.247 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.247 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.247 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.247 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.247 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.248 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.353951_157.1", 2020-11-29T16:11:13.248 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.248 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27", 2020-11-29T16:11:13.248 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.353951918Z", 2020-11-29T16:11:13.248 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.249 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.249 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.249 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.249 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.249 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.249 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Wc4auwc30IepX657G2eZM-u", 2020-11-29T16:11:13.250 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.250 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.250 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.250 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.251 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.251 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.251 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.251 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.251 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.390106_158.1", 2020-11-29T16:11:13.252 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.252 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:13.252 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.390106865Z", 2020-11-29T16:11:13.252 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.252 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.253 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.253 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.253 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.253 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.253 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:13.253 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.254 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.254 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.254 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.254 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.254 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.255 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.255 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.255 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.397881_159.1", 2020-11-29T16:11:13.255 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.255 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:13.256 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.397881409Z", 2020-11-29T16:11:13.256 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.256 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.256 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.256 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.256 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.257 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.257 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:13.257 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.257 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.257 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.258 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.258 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.258 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.258 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.258 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.259 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.400119_160.1", 2020-11-29T16:11:13.259 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.259 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28", 2020-11-29T16:11:13.259 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.400119169Z", 2020-11-29T16:11:13.259 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.259 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.260 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.260 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.260 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.260 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.260 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vorB3p52YuzxIbSF-GHscIz", 2020-11-29T16:11:13.261 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.261 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.261 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.261 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.261 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.262 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.262 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.262 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.262 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.404155_161.1", 2020-11-29T16:11:13.262 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.263 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28", 2020-11-29T16:11:13.263 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.404155690Z", 2020-11-29T16:11:13.263 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.263 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.263 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.264 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.264 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.264 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.264 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_vorB3p52YuzxIbSF-GHscIz", 2020-11-29T16:11:13.264 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.265 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.265 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.265 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.265 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.265 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.266 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.266 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.266 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.461726_162.1", 2020-11-29T16:11:13.266 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.266 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:13.267 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.461726137Z", 2020-11-29T16:11:13.267 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.267 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.267 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.268 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.268 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.268 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.268 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:13.268 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.269 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.269 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.269 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.269 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.270 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.270 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.270 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.270 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.466239_163.1", 2020-11-29T16:11:13.270 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.271 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:13.271 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.466239727Z", 2020-11-29T16:11:13.271 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.271 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.272 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.272 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.272 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.272 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.272 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:13.273 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.273 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.273 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.273 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.273 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.274 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.274 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.274 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.274 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.471274_164.1", 2020-11-29T16:11:13.274 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.275 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29", 2020-11-29T16:11:13.275 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.471274792Z", 2020-11-29T16:11:13.275 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.275 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.275 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.276 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.276 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.276 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.276 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tW9vOLbEcF0UPbuJYTP9PYC", 2020-11-29T16:11:13.276 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.277 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.277 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.277 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.277 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.277 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.278 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.278 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.278 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.475990_165.1", 2020-11-29T16:11:13.278 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.278 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29", 2020-11-29T16:11:13.279 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.475990005Z", 2020-11-29T16:11:13.279 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.279 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.279 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.279 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.280 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.280 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.280 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tW9vOLbEcF0UPbuJYTP9PYC", 2020-11-29T16:11:13.280 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.280 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.281 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.281 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.281 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.281 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.281 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.282 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.282 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.500752_166.1", 2020-11-29T16:11:13.282 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.282 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:13.282 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.500752451Z", 2020-11-29T16:11:13.283 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.283 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.283 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:13.283 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.283 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.283 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.284 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:13.284 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.284 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.284 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.284 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.285 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.285 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.285 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.285 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.285 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.506749_167.1", 2020-11-29T16:11:13.285 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.286 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:13.286 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.506749250Z", 2020-11-29T16:11:13.286 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.286 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.286 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:13.287 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.287 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.287 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.287 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:13.287 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.288 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.288 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.288 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.288 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.288 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.288 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.289 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.289 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.561039_168.1", 2020-11-29T16:11:13.289 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.289 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:13.289 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.561039789Z", 2020-11-29T16:11:13.290 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.290 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.290 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:13.290 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.290 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.291 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.291 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:13.291 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.291 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.291 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.292 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.292 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.292 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.292 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.292 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.293 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.568104_169.1", 2020-11-29T16:11:13.293 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.293 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:13.293 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.568104840Z", 2020-11-29T16:11:13.293 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.294 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.294 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:13.294 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.294 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.294 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.295 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:13.295 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.295 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.295 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.295 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.295 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.296 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.296 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.296 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.296 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.624595_170.1", 2020-11-29T16:11:13.296 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.297 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:13.297 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.624595921Z", 2020-11-29T16:11:13.297 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.297 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.297 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:13.298 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.298 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.298 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.298 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:13.298 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.299 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.299 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.299 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.299 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.299 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.299 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.300 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.300 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.631524_171.1", 2020-11-29T16:11:13.300 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.300 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:13.301 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.631524395Z", 2020-11-29T16:11:13.301 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.301 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.301 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:13.301 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.302 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.302 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.302 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:13.302 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.302 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.303 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.303 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.303 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.303 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.303 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.304 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.304 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.709256_172.1", 2020-11-29T16:11:13.304 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.304 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:13.304 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.709256940Z", 2020-11-29T16:11:13.305 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.305 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.305 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.305 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.305 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.306 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.306 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:13.306 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.306 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.306 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.307 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.307 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.307 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.307 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.307 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.308 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.713773_173.1", 2020-11-29T16:11:13.308 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.308 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:13.308 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.713773728Z", 2020-11-29T16:11:13.308 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.309 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.309 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.309 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.309 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.309 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.310 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:13.310 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.310 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.310 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.311 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.311 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.311 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.311 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.311 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.312 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.718379_174.1", 2020-11-29T16:11:13.312 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.312 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30", 2020-11-29T16:11:13.312 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.718379824Z", 2020-11-29T16:11:13.313 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.313 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.313 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.313 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.313 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.314 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.314 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_h_WBzhTZe664PHorKv6_Am0", 2020-11-29T16:11:13.314 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.314 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.315 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.315 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.315 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.315 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.315 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.316 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.316 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.723686_175.1", 2020-11-29T16:11:13.316 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.316 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30", 2020-11-29T16:11:13.317 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.723686356Z", 2020-11-29T16:11:13.317 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.317 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.317 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.317 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.318 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.318 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.318 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_h_WBzhTZe664PHorKv6_Am0", 2020-11-29T16:11:13.318 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.319 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.319 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.319 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.319 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.319 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.320 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.320 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.320 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.752571_176.1", 2020-11-29T16:11:13.320 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.321 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:13.321 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.752571930Z", 2020-11-29T16:11:13.321 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.321 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.321 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.322 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.322 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.322 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.322 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:13.323 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.323 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.323 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.323 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.324 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.324 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.324 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.324 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.325 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.756904_177.1", 2020-11-29T16:11:13.325 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.325 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:13.325 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.756904681Z", 2020-11-29T16:11:13.326 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.326 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.326 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.327 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.327 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.327 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.327 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:13.327 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.328 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.328 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.328 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.328 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.328 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.329 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.329 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.329 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.759821_178.1", 2020-11-29T16:11:13.329 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.329 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31", 2020-11-29T16:11:13.330 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.759821922Z", 2020-11-29T16:11:13.330 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.330 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.330 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.330 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.331 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.331 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.331 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_eKjXcKGmg5yyLHdihUsR2sW", 2020-11-29T16:11:13.331 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.331 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.332 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.332 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.332 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.332 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.332 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.333 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.333 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.764291_179.1", 2020-11-29T16:11:13.333 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.333 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31", 2020-11-29T16:11:13.333 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.764291393Z", 2020-11-29T16:11:13.334 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.334 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.334 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.334 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.334 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.335 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.335 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_eKjXcKGmg5yyLHdihUsR2sW", 2020-11-29T16:11:13.335 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.335 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.335 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.336 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.336 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.336 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.336 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.336 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.336 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.825137_180.1", 2020-11-29T16:11:13.337 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.337 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:13.337 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.825137534Z", 2020-11-29T16:11:13.337 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.337 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.338 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.338 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.338 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.338 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.338 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:13.339 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.339 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.339 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.339 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.339 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.340 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.340 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.340 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.340 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.829684_181.1", 2020-11-29T16:11:13.341 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.341 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:13.341 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.829684939Z", 2020-11-29T16:11:13.341 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.341 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.341 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.342 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.342 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.342 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.342 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:13.342 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.343 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.343 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.343 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.343 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.343 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.344 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.344 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.344 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.831781_182.1", 2020-11-29T16:11:13.344 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.344 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32", 2020-11-29T16:11:13.344 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.831781072Z", 2020-11-29T16:11:13.345 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.345 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.345 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.345 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.345 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.346 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.346 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_dhvj1N7nCxk5qukpVfy3IjW", 2020-11-29T16:11:13.346 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.346 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.346 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.347 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.347 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.347 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.347 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.347 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.348 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.835966_183.1", 2020-11-29T16:11:13.348 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.348 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32", 2020-11-29T16:11:13.348 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.835966939Z", 2020-11-29T16:11:13.348 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.349 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.349 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.349 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.349 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.349 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.350 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_dhvj1N7nCxk5qukpVfy3IjW", 2020-11-29T16:11:13.350 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.350 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.350 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.350 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.351 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.351 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.351 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.351 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.351 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.860087_184.1", 2020-11-29T16:11:13.352 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.352 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:13.352 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.860087065Z", 2020-11-29T16:11:13.352 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.352 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.353 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:13.353 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.353 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.353 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.354 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:13.354 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.354 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.354 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.354 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.355 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.355 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.355 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.355 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.355 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.865870_185.1", 2020-11-29T16:11:13.356 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.356 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:13.356 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.865870483Z", 2020-11-29T16:11:13.356 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.356 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.357 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:13.357 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.357 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.357 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.357 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:13.358 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.358 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.358 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.358 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.359 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.359 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.359 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.359 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.359 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.914709_186.1", 2020-11-29T16:11:13.360 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.360 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:13.360 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.914709667Z", 2020-11-29T16:11:13.360 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.360 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.361 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:13.361 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.361 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.361 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.361 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:13.362 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.362 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.362 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.362 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.362 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.363 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.363 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.363 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.363 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.925154_187.1", 2020-11-29T16:11:13.363 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.364 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:13.364 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.925154339Z", 2020-11-29T16:11:13.364 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.364 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.365 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:13.365 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.365 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.365 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.365 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:13.366 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.366 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.366 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.366 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.366 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.367 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.367 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.367 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.367 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.991321_188.1", 2020-11-29T16:11:13.367 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.368 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:13.368 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.991321647Z", 2020-11-29T16:11:13.368 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.368 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.368 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:13.369 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.369 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.369 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.369 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:13.369 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.370 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.370 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.370 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.370 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.370 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.371 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.371 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.371 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666122.999983_189.1", 2020-11-29T16:11:13.371 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.371 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:13.372 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:42.999983004Z", 2020-11-29T16:11:13.372 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.372 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.372 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:13.372 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.372 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.373 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.373 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:13.373 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.373 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.373 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.374 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.374 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.374 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.374 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.374 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.375 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.076605_190.1", 2020-11-29T16:11:13.375 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.375 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:13.375 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.076605830Z", 2020-11-29T16:11:13.375 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.376 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.376 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.376 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.376 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.377 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.377 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:13.377 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.377 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.377 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.377 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.378 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.378 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.378 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.378 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.379 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.081952_191.1", 2020-11-29T16:11:13.379 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.379 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:13.379 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.081952008Z", 2020-11-29T16:11:13.379 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.380 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.380 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.380 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.380 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.380 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.380 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.381 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.382 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.382 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.382 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.382 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.083820_192.1", 2020-11-29T16:11:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33", 2020-11-29T16:11:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.083820333Z", 2020-11-29T16:11:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.383 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_aScNky5UrTuH12xWR79F83e", 2020-11-29T16:11:13.384 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.385 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.088982_193.1", 2020-11-29T16:11:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.386 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33", 2020-11-29T16:11:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.088982559Z", 2020-11-29T16:11:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.387 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_aScNky5UrTuH12xWR79F83e", 2020-11-29T16:11:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.388 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.389 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.389 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.389 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.389 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.389 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.390 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.121086_194.1", 2020-11-29T16:11:13.390 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.390 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:13.390 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.121086133Z", 2020-11-29T16:11:13.390 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.391 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.391 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.391 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.391 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.391 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.392 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:13.392 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.392 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.392 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.392 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.393 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.393 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.393 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.393 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.393 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.133205_195.1", 2020-11-29T16:11:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.133205341Z", 2020-11-29T16:11:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.394 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:13.395 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.396 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.397 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.397 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.397 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.138006_196.1", 2020-11-29T16:11:13.397 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.397 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34", 2020-11-29T16:11:13.397 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.138006291Z", 2020-11-29T16:11:13.398 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.398 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.398 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.398 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.398 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_9r_K5jlLAncl40ETfoVeJjK", 2020-11-29T16:11:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.153529_197.1", 2020-11-29T16:11:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34", 2020-11-29T16:11:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.153529708Z", 2020-11-29T16:11:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_9r_K5jlLAncl40ETfoVeJjK", 2020-11-29T16:11:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.225476_198.1", 2020-11-29T16:11:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.225476965Z", 2020-11-29T16:11:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.230332_199.1", 2020-11-29T16:11:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.230332153Z", 2020-11-29T16:11:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.411 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.411 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.411 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.411 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.411 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.412 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.412 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.412 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.232895_200.1", 2020-11-29T16:11:13.412 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.412 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35", 2020-11-29T16:11:13.412 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.232895662Z", 2020-11-29T16:11:13.413 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.413 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.413 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.413 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.414 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.414 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.414 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_BauiNjhDLe4JD_SZUYzW5BH", 2020-11-29T16:11:13.414 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.414 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.414 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.415 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.415 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.415 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.415 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.415 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.416 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.237930_201.1", 2020-11-29T16:11:13.416 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.416 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35", 2020-11-29T16:11:13.416 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.237930233Z", 2020-11-29T16:11:13.417 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.417 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.417 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.417 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.417 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.418 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.418 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_BauiNjhDLe4JD_SZUYzW5BH", 2020-11-29T16:11:13.418 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.418 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.418 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.419 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.419 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.419 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.419 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.419 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.420 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.261144_202.1", 2020-11-29T16:11:13.420 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.420 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:13.420 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.261144444Z", 2020-11-29T16:11:13.420 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.421 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.421 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:13.421 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.421 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.421 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.422 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:13.422 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.422 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.422 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.422 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.423 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.423 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.423 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.423 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.423 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666123.266462_203.1", 2020-11-29T16:11:13.424 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.424 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:13.424 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:43.266462524Z", 2020-11-29T16:11:13.424 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.424 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.425 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:13.425 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.425 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.425 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.425 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:13.426 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.426 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.426 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.426 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.427 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.427 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.427 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.427 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.427 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666124.313715_204.1", 2020-11-29T16:11:13.428 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.428 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:13.428 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:44.313715914Z", 2020-11-29T16:11:13.428 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.428 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.429 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:13.429 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.429 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.429 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.430 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:13.430 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.430 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.430 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.430 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.431 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.431 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.431 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.431 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.431 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666124.327834_205.1", 2020-11-29T16:11:13.432 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.432 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:13.432 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:44.327834836Z", 2020-11-29T16:11:13.432 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.432 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.433 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:13.433 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.433 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.433 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.433 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:13.434 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.434 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.434 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.434 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.434 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.435 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.435 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.435 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.435 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.389734_206.1", 2020-11-29T16:11:13.436 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.436 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:13.436 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.389734131Z", 2020-11-29T16:11:13.436 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.436 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.437 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:13.437 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.437 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.437 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.437 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:13.438 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.438 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.438 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.438 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.438 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.439 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.439 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.439 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.439 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.399959_207.1", 2020-11-29T16:11:13.439 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.440 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:13.440 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.399959602Z", 2020-11-29T16:11:13.440 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.440 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.441 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:13.441 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.441 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.441 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.441 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:13.442 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 2 2020-11-29T16:11:13.442 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.442 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.442 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.442 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.443 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.443 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.443 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.443 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.480704_208.1", 2020-11-29T16:11:13.443 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.444 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36", 2020-11-29T16:11:13.444 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.480704401Z", 2020-11-29T16:11:13.444 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.444 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.444 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.445 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.445 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.445 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.445 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_UJxOrACWaVFo5tZbkaJd5aB", 2020-11-29T16:11:13.445 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.446 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.446 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.446 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.446 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.446 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.447 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.447 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.447 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.485309_209.1", 2020-11-29T16:11:13.447 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.447 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36", 2020-11-29T16:11:13.448 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.485309628Z", 2020-11-29T16:11:13.448 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.448 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.448 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.448 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.448 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.449 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.449 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_UJxOrACWaVFo5tZbkaJd5aB", 2020-11-29T16:11:13.449 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.449 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.449 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.450 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.450 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.450 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.450 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.450 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.451 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.487975_210.1", 2020-11-29T16:11:13.451 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.451 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-36", 2020-11-29T16:11:13.451 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.487975960Z", 2020-11-29T16:11:13.451 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.452 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.452 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.452 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.452 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.452 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.453 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VZJxPfFqHBrvBon1FcGTxPs", 2020-11-29T16:11:13.453 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.453 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.453 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.453 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.454 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.454 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.454 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.454 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.454 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.491713_211.1", 2020-11-29T16:11:13.455 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.455 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-36", 2020-11-29T16:11:13.455 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.491713665Z", 2020-11-29T16:11:13.455 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.455 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.456 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.456 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.456 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.456 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.456 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_VZJxPfFqHBrvBon1FcGTxPs", 2020-11-29T16:11:13.456 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.457 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.457 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.457 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.457 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.457 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.458 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.458 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.458 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.525177_212.1", 2020-11-29T16:11:13.458 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.458 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37", 2020-11-29T16:11:13.459 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.525177090Z", 2020-11-29T16:11:13.459 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.459 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.459 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.459 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.460 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.460 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.460 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_syk3UA8fSzEvnh7yGHxS7iU", 2020-11-29T16:11:13.460 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.460 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.461 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.461 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.461 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.461 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.461 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.462 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.462 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.534079_213.1", 2020-11-29T16:11:13.462 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.462 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37", 2020-11-29T16:11:13.462 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.534079771Z", 2020-11-29T16:11:13.463 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.463 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.463 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.463 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.463 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.464 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.464 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_syk3UA8fSzEvnh7yGHxS7iU", 2020-11-29T16:11:13.464 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.464 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.464 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.465 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.465 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.465 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.465 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.465 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.466 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.540941_214.1", 2020-11-29T16:11:13.466 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.466 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-37", 2020-11-29T16:11:13.466 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.540941786Z", 2020-11-29T16:11:13.467 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.467 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.467 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.467 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.467 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.467 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.468 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_LPLM1eQYX53fxff8emhXqjO", 2020-11-29T16:11:13.468 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.468 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.468 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.468 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.469 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.469 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.469 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.469 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.469 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.547825_215.1", 2020-11-29T16:11:13.470 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.470 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-37", 2020-11-29T16:11:13.470 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.547825214Z", 2020-11-29T16:11:13.470 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.471 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.471 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.471 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.471 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.471 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.472 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_LPLM1eQYX53fxff8emhXqjO", 2020-11-29T16:11:13.472 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.472 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.472 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.472 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.473 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.473 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.473 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.473 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.473 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.590883_216.1", 2020-11-29T16:11:13.474 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.474 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38", 2020-11-29T16:11:13.474 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.590883464Z", 2020-11-29T16:11:13.474 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.474 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.475 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.475 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.475 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.475 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.475 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_sluiDwZnW0jlavok--xXUQM", 2020-11-29T16:11:13.476 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.476 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.476 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.476 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.476 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.477 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.477 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.477 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.477 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.595783_217.1", 2020-11-29T16:11:13.478 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.478 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38", 2020-11-29T16:11:13.478 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.595783563Z", 2020-11-29T16:11:13.478 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.478 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.479 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.479 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.479 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.479 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.479 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_sluiDwZnW0jlavok--xXUQM", 2020-11-29T16:11:13.480 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.480 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.480 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.480 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.480 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.481 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.481 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.481 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.481 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.597926_218.1", 2020-11-29T16:11:13.481 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.482 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-38", 2020-11-29T16:11:13.482 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.597926148Z", 2020-11-29T16:11:13.482 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.482 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.482 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.483 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.483 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.483 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.483 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_s5CCY0lZsRXaG97sebP-L5L", 2020-11-29T16:11:13.483 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.484 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.484 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.484 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.484 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.484 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.485 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.485 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.485 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666125.602582_219.1", 2020-11-29T16:11:13.485 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.485 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-38", 2020-11-29T16:11:13.485 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:45.602582602Z", 2020-11-29T16:11:13.486 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.486 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.486 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.486 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.486 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.487 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.487 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_s5CCY0lZsRXaG97sebP-L5L", 2020-11-29T16:11:13.487 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.487 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.487 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.488 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.488 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.488 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.488 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.488 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.489 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.333843_222.1", 2020-11-29T16:11:13.489 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.489 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-39:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39", 2020-11-29T16:11:13.489 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.333843820Z", 2020-11-29T16:11:13.490 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.490 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.490 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.490 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.490 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.491 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.491 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5H0pP9yCjtoMl1W4TvUgcb8", 2020-11-29T16:11:13.491 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.491 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.491 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.492 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.492 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.492 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.492 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.492 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.493 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.338434_223.1", 2020-11-29T16:11:13.493 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.493 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-39:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39", 2020-11-29T16:11:13.493 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.338434096Z", 2020-11-29T16:11:13.493 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.494 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.494 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.494 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.494 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.494 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.495 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5H0pP9yCjtoMl1W4TvUgcb8", 2020-11-29T16:11:13.495 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.495 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.495 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.495 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.496 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.496 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.496 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.496 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.496 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.340224_224.1", 2020-11-29T16:11:13.497 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.497 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-39", 2020-11-29T16:11:13.497 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.340224571Z", 2020-11-29T16:11:13.497 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.497 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.498 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.498 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.498 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.498 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.499 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Pp8cXl7djO1gxLhMq2AZXJp", 2020-11-29T16:11:13.499 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.499 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.499 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.499 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.500 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.500 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.500 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.500 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.500 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.343737_225.1", 2020-11-29T16:11:13.501 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.501 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-39", 2020-11-29T16:11:13.501 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.343737416Z", 2020-11-29T16:11:13.501 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.501 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.502 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.502 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.502 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.502 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.503 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_Pp8cXl7djO1gxLhMq2AZXJp", 2020-11-29T16:11:13.503 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.503 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.503 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.503 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.504 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.504 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.504 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.504 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.504 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.368575_226.1", 2020-11-29T16:11:13.505 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.505 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-40:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40", 2020-11-29T16:11:13.505 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.368575114Z", 2020-11-29T16:11:13.505 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.506 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.506 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.506 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.506 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.506 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.507 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jgn-ZwFyVxv-tmED-Ggd4Hw", 2020-11-29T16:11:13.507 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.507 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.507 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.507 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.508 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.508 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.508 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.508 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.508 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.374510_227.1", 2020-11-29T16:11:13.509 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.509 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-40:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40", 2020-11-29T16:11:13.509 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.374510224Z", 2020-11-29T16:11:13.509 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.509 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.510 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.510 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.510 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.510 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.510 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_jgn-ZwFyVxv-tmED-Ggd4Hw", 2020-11-29T16:11:13.511 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.511 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.511 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.511 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.511 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.512 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.512 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.512 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.512 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.376835_228.1", 2020-11-29T16:11:13.513 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.513 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-40", 2020-11-29T16:11:13.513 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.376835132Z", 2020-11-29T16:11:13.513 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.513 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.514 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.514 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.514 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.514 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.514 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_qrn1nKXKg0aZIU2cqKpKgju", 2020-11-29T16:11:13.515 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.515 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.515 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.515 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.515 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.516 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.516 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.516 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.516 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.380733_229.1", 2020-11-29T16:11:13.517 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.517 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-40", 2020-11-29T16:11:13.517 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.380733954Z", 2020-11-29T16:11:13.517 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.517 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.517 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.518 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.518 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.518 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.518 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_qrn1nKXKg0aZIU2cqKpKgju", 2020-11-29T16:11:13.518 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.519 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.519 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.519 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.519 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.520 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.520 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.520 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.520 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.430813_231.1", 2020-11-29T16:11:13.520 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.521 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-41:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41", 2020-11-29T16:11:13.521 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.430813301Z", 2020-11-29T16:11:13.521 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.521 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.521 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.522 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.522 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.522 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.522 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_upM-2-zq2ujNZABi5Lo2I6j", 2020-11-29T16:11:13.522 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.523 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.523 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.523 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.523 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.523 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.524 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.524 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.524 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.440999_232.1", 2020-11-29T16:11:13.524 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.524 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-41:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41", 2020-11-29T16:11:13.525 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.440999402Z", 2020-11-29T16:11:13.525 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.525 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.525 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.525 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.526 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.526 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.526 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_upM-2-zq2ujNZABi5Lo2I6j", 2020-11-29T16:11:13.526 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.526 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.527 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.527 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.527 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.527 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.528 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.528 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.528 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.444219_234.1", 2020-11-29T16:11:13.528 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.528 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-41", 2020-11-29T16:11:13.529 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.444219056Z", 2020-11-29T16:11:13.529 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.529 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.529 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.529 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.530 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.530 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.530 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_iwxqSbv55pvq-WwQMf1oR3e", 2020-11-29T16:11:13.530 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.530 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.531 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.531 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:13.531 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.531 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.531 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.532 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:13.532 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666127.452875_235.1", 2020-11-29T16:11:13.532 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:13.532 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-41", 2020-11-29T16:11:13.532 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.452875466Z", 2020-11-29T16:11:13.533 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:13.533 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:13.533 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:13.533 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:13.533 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.533 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:13.534 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_iwxqSbv55pvq-WwQMf1oR3e", 2020-11-29T16:11:13.534 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:13.534 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:13.534 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:13.534 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:13.535 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.535 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.535 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:13.535 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:13.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:13.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:13.538 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e24cddf9-396c-4261-a7bc-b4e1e678325e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:13.576 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:13.576 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:13.814 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:13.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.815 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665466.760427_46.1", 2020-11-29T16:11:13.815 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "user", 2020-11-29T16:11:13.815 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "rgw-multisite-test-user", 2020-11-29T16:11:13.815 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:57:46.760427145Z", 2020-11-29T16:11:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.816 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7jrnGFazjfCMV9OgOp-5ICR", 2020-11-29T16:11:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.818 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.818 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.819 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665466.770600_48.1", 2020-11-29T16:11:13.819 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "user", 2020-11-29T16:11:13.819 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "rgw-multisite-test-user", 2020-11-29T16:11:13.820 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:57:46.770600943Z", 2020-11-29T16:11:13.820 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.820 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.820 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.820 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.821 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.821 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7jrnGFazjfCMV9OgOp-5ICR", 2020-11-29T16:11:13.821 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.822 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.822 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.823 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.309764_49.1", 2020-11-29T16:11:13.823 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.823 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-1:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1", 2020-11-29T16:11:13.823 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.309764005Z", 2020-11-29T16:11:13.824 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.824 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.824 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.824 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.825 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.825 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tDx7cRFyPylOsARNJShhjdi", 2020-11-29T16:11:13.825 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.825 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.826 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.827 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.329195_52.1", 2020-11-29T16:11:13.827 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.827 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-1:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1", 2020-11-29T16:11:13.827 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.329195289Z", 2020-11-29T16:11:13.827 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.828 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.828 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.828 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.829 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.829 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tDx7cRFyPylOsARNJShhjdi", 2020-11-29T16:11:13.829 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.829 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.830 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.330857_53.1", 2020-11-29T16:11:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-1", 2020-11-29T16:11:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.330857588Z", 2020-11-29T16:11:13.831 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.832 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.832 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.832 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.833 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_da31gfzBeEr_qTz8sD5Z95W", 2020-11-29T16:11:13.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.833 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.834 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.835 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.334069_54.1", 2020-11-29T16:11:13.835 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.835 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-1", 2020-11-29T16:11:13.835 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.334069131Z", 2020-11-29T16:11:13.835 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.836 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.836 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.836 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.837 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.837 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_da31gfzBeEr_qTz8sD5Z95W", 2020-11-29T16:11:13.837 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.837 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.838 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.839 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.361540_55.1", 2020-11-29T16:11:13.839 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.839 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-2:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2", 2020-11-29T16:11:13.839 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.361540407Z", 2020-11-29T16:11:13.839 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.840 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.840 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.840 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.840 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.841 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jcxJvuSk8NgQdLMU8LRCoaP", 2020-11-29T16:11:13.841 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.841 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.841 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.842 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.366215_56.1", 2020-11-29T16:11:13.843 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.843 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-2:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2", 2020-11-29T16:11:13.843 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.366215003Z", 2020-11-29T16:11:13.843 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.844 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.844 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.844 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.845 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jcxJvuSk8NgQdLMU8LRCoaP", 2020-11-29T16:11:13.845 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.845 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.845 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.846 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.369096_57.1", 2020-11-29T16:11:13.847 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.847 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-2", 2020-11-29T16:11:13.847 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.369096676Z", 2020-11-29T16:11:13.847 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.847 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.848 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.848 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.848 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.848 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_F0nqKIPOuZ20dNAraH9mMFO", 2020-11-29T16:11:13.849 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.849 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.849 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.850 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665590.374355_58.1", 2020-11-29T16:11:13.850 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.851 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-2", 2020-11-29T16:11:13.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:50.374355442Z", 2020-11-29T16:11:13.851 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.851 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.851 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.852 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.852 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.852 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_F0nqKIPOuZ20dNAraH9mMFO", 2020-11-29T16:11:13.852 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.853 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.853 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.854 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665593.914359_59.1", 2020-11-29T16:11:13.854 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.855 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-3:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3", 2020-11-29T16:11:13.855 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.914359282Z", 2020-11-29T16:11:13.855 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.855 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.855 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.856 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.856 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_nPQIea0SAu2YbHSwS1hqdTS", 2020-11-29T16:11:13.856 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.857 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.858 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665593.919600_60.1", 2020-11-29T16:11:13.858 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.858 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-3:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3", 2020-11-29T16:11:13.859 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.919600334Z", 2020-11-29T16:11:13.859 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.859 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.859 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.859 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.860 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.860 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_nPQIea0SAu2YbHSwS1hqdTS", 2020-11-29T16:11:13.860 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.860 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.861 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.862 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665593.924502_61.1", 2020-11-29T16:11:13.862 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.862 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-3", 2020-11-29T16:11:13.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.924502922Z", 2020-11-29T16:11:13.862 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.863 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.863 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.863 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.864 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hgxnPf55gFvOJXZoH0wMw6D", 2020-11-29T16:11:13.864 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.864 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.864 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.865 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665593.929811_62.1", 2020-11-29T16:11:13.866 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.866 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-3", 2020-11-29T16:11:13.866 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:53.929811866Z", 2020-11-29T16:11:13.866 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.866 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.867 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.867 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.868 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hgxnPf55gFvOJXZoH0wMw6D", 2020-11-29T16:11:13.868 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.868 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.868 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.869 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.255794_63.1", 2020-11-29T16:11:13.869 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.870 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-4:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4", 2020-11-29T16:11:13.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.255794146Z", 2020-11-29T16:11:13.870 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.870 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.870 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dI-T3JKYYOksP3k9JHqk_4S", 2020-11-29T16:11:13.871 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.873 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.260821_65.1", 2020-11-29T16:11:13.873 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.873 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-4:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4", 2020-11-29T16:11:13.874 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.260821437Z", 2020-11-29T16:11:13.874 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.874 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.874 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.875 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.875 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dI-T3JKYYOksP3k9JHqk_4S", 2020-11-29T16:11:13.875 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.876 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.876 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.877 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.262726_66.1", 2020-11-29T16:11:13.877 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.877 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-4", 2020-11-29T16:11:13.877 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.262726231Z", 2020-11-29T16:11:13.878 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.878 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.878 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.878 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.878 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.879 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_H0hQB2qGLgINO4xvzkf-CBV", 2020-11-29T16:11:13.879 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.879 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.879 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.880 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.880 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.880 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.267845_67.1", 2020-11-29T16:11:13.881 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.881 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-4", 2020-11-29T16:11:13.881 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.267845461Z", 2020-11-29T16:11:13.881 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.882 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.882 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.882 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.882 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.882 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_H0hQB2qGLgINO4xvzkf-CBV", 2020-11-29T16:11:13.883 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.883 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.883 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.884 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.291993_68.1", 2020-11-29T16:11:13.885 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.885 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-5:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5", 2020-11-29T16:11:13.885 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.291993232Z", 2020-11-29T16:11:13.885 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.885 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.886 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.886 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.886 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.886 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_eNmUNSw8O2Q1UqGf3-a8M0y", 2020-11-29T16:11:13.886 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.887 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.887 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.888 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.297549_69.1", 2020-11-29T16:11:13.888 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.888 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-5:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5", 2020-11-29T16:11:13.889 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.297549197Z", 2020-11-29T16:11:13.889 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.889 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.889 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.889 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_eNmUNSw8O2Q1UqGf3-a8M0y", 2020-11-29T16:11:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.890 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.891 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.299887_70.1", 2020-11-29T16:11:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-5", 2020-11-29T16:11:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.299887736Z", 2020-11-29T16:11:13.892 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.893 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_axHo0yOC2VpaRAyEFVHuReN", 2020-11-29T16:11:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.894 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.895 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.307514_71.1", 2020-11-29T16:11:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-5", 2020-11-29T16:11:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.307514021Z", 2020-11-29T16:11:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.896 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_axHo0yOC2VpaRAyEFVHuReN", 2020-11-29T16:11:13.897 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.352257_72.1", 2020-11-29T16:11:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.899 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-6:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6", 2020-11-29T16:11:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.352257512Z", 2020-11-29T16:11:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_nbxbaUi2Y7VYrWoa72raC5j", 2020-11-29T16:11:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.901 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.359127_73.1", 2020-11-29T16:11:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-6:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6", 2020-11-29T16:11:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.359127428Z", 2020-11-29T16:11:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_nbxbaUi2Y7VYrWoa72raC5j", 2020-11-29T16:11:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.361298_74.1", 2020-11-29T16:11:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-6", 2020-11-29T16:11:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.361298888Z", 2020-11-29T16:11:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5-Sqi-mEXlqXjQFGM13_N21", 2020-11-29T16:11:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665595.365399_75.1", 2020-11-29T16:11:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-6", 2020-11-29T16:11:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:55.365399707Z", 2020-11-29T16:11:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5-Sqi-mEXlqXjQFGM13_N21", 2020-11-29T16:11:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.250797_76.1", 2020-11-29T16:11:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7", 2020-11-29T16:11:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.250797689Z", 2020-11-29T16:11:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_T6kPv-3jveF29kZeCLJlo-6", 2020-11-29T16:11:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.255494_79.1", 2020-11-29T16:11:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7", 2020-11-29T16:11:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.255494488Z", 2020-11-29T16:11:13.918 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.919 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_T6kPv-3jveF29kZeCLJlo-6", 2020-11-29T16:11:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.920 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.257909_80.1", 2020-11-29T16:11:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7", 2020-11-29T16:11:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.257909992Z", 2020-11-29T16:11:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.265880_81.1", 2020-11-29T16:11:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7", 2020-11-29T16:11:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.265880801Z", 2020-11-29T16:11:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.926 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.284752_82.1", 2020-11-29T16:11:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8", 2020-11-29T16:11:13.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.284752329Z", 2020-11-29T16:11:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Qg8VBYmtvIuXfPsbR8FJmF5", 2020-11-29T16:11:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.931 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.290062_83.1", 2020-11-29T16:11:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8", 2020-11-29T16:11:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.290062502Z", 2020-11-29T16:11:13.933 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.934 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Qg8VBYmtvIuXfPsbR8FJmF5", 2020-11-29T16:11:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.292190_84.1", 2020-11-29T16:11:13.936 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8", 2020-11-29T16:11:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.292190927Z", 2020-11-29T16:11:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.937 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:13.938 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.296685_85.1", 2020-11-29T16:11:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8", 2020-11-29T16:11:13.940 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.296685079Z", 2020-11-29T16:11:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.942 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.942 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:13.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.942 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.943 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.944 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.338804_86.1", 2020-11-29T16:11:13.944 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.944 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9", 2020-11-29T16:11:13.944 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.338804685Z", 2020-11-29T16:11:13.944 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.944 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.945 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.945 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.945 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.945 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_x_puG7-2t3lKG0zWJ2e3lY4", 2020-11-29T16:11:13.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.946 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.946 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.946 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.946 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.947 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.345806_87.1", 2020-11-29T16:11:13.947 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.947 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9", 2020-11-29T16:11:13.948 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.345806856Z", 2020-11-29T16:11:13.948 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.948 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.948 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.948 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.949 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.949 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_x_puG7-2t3lKG0zWJ2e3lY4", 2020-11-29T16:11:13.949 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.949 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.950 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.951 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.348268_88.1", 2020-11-29T16:11:13.951 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.951 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9", 2020-11-29T16:11:13.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.348268690Z", 2020-11-29T16:11:13.951 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.951 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.952 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.952 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.952 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:13.953 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.953 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.953 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.954 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665598.354068_89.1", 2020-11-29T16:11:13.954 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.955 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9", 2020-11-29T16:11:13.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:58.354068281Z", 2020-11-29T16:11:13.955 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.955 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.955 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.956 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.956 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.956 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:13.956 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.957 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.957 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.958 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.682590_90.1", 2020-11-29T16:11:13.958 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.958 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7", 2020-11-29T16:11:13.958 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.682590411Z", 2020-11-29T16:11:13.959 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.959 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.959 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:13.959 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.960 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-11-29T16:11:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.961 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.689390_92.1", 2020-11-29T16:11:13.962 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.962 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-7", 2020-11-29T16:11:13.962 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.689390648Z", 2020-11-29T16:11:13.962 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.962 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.962 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kFxvpbHz5bLOENIZMFtgFNf", 2020-11-29T16:11:13.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.964 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.966 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.968 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.764882_93.1", 2020-11-29T16:11:13.968 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.968 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8", 2020-11-29T16:11:13.968 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.764882801Z", 2020-11-29T16:11:13.968 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.969 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.969 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:13.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.969 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.970 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:13.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:13.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.970 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.970 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-11-29T16:11:13.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.971 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.770654_94.1", 2020-11-29T16:11:13.972 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.972 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-8", 2020-11-29T16:11:13.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.770654401Z", 2020-11-29T16:11:13.972 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.972 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_i_EBmJNMErB04LGZl73tPMP", 2020-11-29T16:11:13.973 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.889388_95.1", 2020-11-29T16:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.975 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9", 2020-11-29T16:11:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.889388291Z", 2020-11-29T16:11:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.977 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.977 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:13.977 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:13.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.977 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.978 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-11-29T16:11:13.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.979 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665599.895400_96.1", 2020-11-29T16:11:13.979 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.979 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-9", 2020-11-29T16:11:13.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400726Z", 2020-11-29T16:11:13.979 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.980 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.980 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:13.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.980 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.981 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ABgL14Q8t0t3L3u29heULT0", 2020-11-29T16:11:13.981 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:13.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.981 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.981 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.982 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.982 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.982 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.191369_34.1", 2020-11-29T16:11:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10", 2020-11-29T16:11:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.191369861Z", 2020-11-29T16:11:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.983 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.984 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.984 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.984 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.984 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YrVXr7ERmibtK1P81HX79vZ", 2020-11-29T16:11:13.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.985 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.985 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.986 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.195873_35.1", 2020-11-29T16:11:13.986 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.987 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10", 2020-11-29T16:11:13.987 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.195873130Z", 2020-11-29T16:11:13.987 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.987 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.987 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.988 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.988 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YrVXr7ERmibtK1P81HX79vZ", 2020-11-29T16:11:13.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.989 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.989 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.990 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.198497_36.1", 2020-11-29T16:11:13.990 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.990 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-10", 2020-11-29T16:11:13.990 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.198497073Z", 2020-11-29T16:11:13.991 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.991 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.991 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.991 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.992 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.992 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DFiygE422CgLw5QBztkfzu-", 2020-11-29T16:11:13.992 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.992 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.993 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:13.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.994 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.204454_37.1", 2020-11-29T16:11:13.994 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:13.994 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-10", 2020-11-29T16:11:13.994 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.204454835Z", 2020-11-29T16:11:13.995 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.995 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.995 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.996 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:13.996 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DFiygE422CgLw5QBztkfzu-", 2020-11-29T16:11:13.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:13.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.997 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:13.997 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:13.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:13.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:13.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:13.998 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.229632_38.1", 2020-11-29T16:11:13.998 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:13.998 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11", 2020-11-29T16:11:13.998 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.229632159Z", 2020-11-29T16:11:13.999 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:13.999 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:13.999 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:13.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:13.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.000 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.000 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_20WS3iYLQJHPayxR2Iu8xxa", 2020-11-29T16:11:14.000 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.002 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.234831_39.1", 2020-11-29T16:11:14.002 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.002 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11", 2020-11-29T16:11:14.002 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.234831409Z", 2020-11-29T16:11:14.003 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.003 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.003 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.003 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.004 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.004 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_20WS3iYLQJHPayxR2Iu8xxa", 2020-11-29T16:11:14.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.004 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.005 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.237152_40.1", 2020-11-29T16:11:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-11", 2020-11-29T16:11:14.006 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.237152720Z", 2020-11-29T16:11:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0w2vqVYxXcNZ-dU0Fiwn2qb", 2020-11-29T16:11:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.241905_41.1", 2020-11-29T16:11:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-11", 2020-11-29T16:11:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.241905299Z", 2020-11-29T16:11:14.010 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0w2vqVYxXcNZ-dU0Fiwn2qb", 2020-11-29T16:11:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.012 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.014 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.311228_42.1", 2020-11-29T16:11:14.014 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.014 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12", 2020-11-29T16:11:14.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.311228153Z", 2020-11-29T16:11:14.014 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.015 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.015 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.015 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.015 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.016 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jrQUaYBzA_aLdZ7o_IQP6PE", 2020-11-29T16:11:14.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.016 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.016 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.018 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.320192_43.1", 2020-11-29T16:11:14.018 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.018 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12", 2020-11-29T16:11:14.018 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.320192323Z", 2020-11-29T16:11:14.018 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.019 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.019 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.019 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.020 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jrQUaYBzA_aLdZ7o_IQP6PE", 2020-11-29T16:11:14.020 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.020 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.020 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.022 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.328671_44.1", 2020-11-29T16:11:14.022 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.022 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-12", 2020-11-29T16:11:14.022 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.328671604Z", 2020-11-29T16:11:14.022 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.023 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.023 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.023 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.023 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.024 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5MnB5hm2ISAIj9JGdxV50H9", 2020-11-29T16:11:14.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.024 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.024 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.025 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665601.335813_45.1", 2020-11-29T16:11:14.026 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.026 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-12", 2020-11-29T16:11:14.026 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:01.335813349Z", 2020-11-29T16:11:14.026 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.027 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.027 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.027 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.027 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.028 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5MnB5hm2ISAIj9JGdxV50H9", 2020-11-29T16:11:14.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.028 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.028 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.029 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.841649_46.1", 2020-11-29T16:11:14.030 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.030 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13", 2020-11-29T16:11:14.030 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.841649100Z", 2020-11-29T16:11:14.030 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.030 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.031 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.031 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.031 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.031 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Kqyzq54nxbVSdNcuPFIuxtb", 2020-11-29T16:11:14.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.032 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.032 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.033 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.847258_47.1", 2020-11-29T16:11:14.033 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.034 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13", 2020-11-29T16:11:14.034 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.847258533Z", 2020-11-29T16:11:14.034 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.034 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.035 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.035 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.035 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.035 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Kqyzq54nxbVSdNcuPFIuxtb", 2020-11-29T16:11:14.036 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.036 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.036 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.036 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.037 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.037 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.849488_48.1", 2020-11-29T16:11:14.037 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.038 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-13", 2020-11-29T16:11:14.038 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.849488010Z", 2020-11-29T16:11:14.038 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.038 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.038 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.039 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rjVcqahbW1Rw3RINe760YzT", 2020-11-29T16:11:14.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.040 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.040 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.041 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.041 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.853552_49.1", 2020-11-29T16:11:14.041 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.042 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-13", 2020-11-29T16:11:14.042 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.853552156Z", 2020-11-29T16:11:14.042 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.042 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.042 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.043 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.043 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rjVcqahbW1Rw3RINe760YzT", 2020-11-29T16:11:14.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.876908_50.1", 2020-11-29T16:11:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14", 2020-11-29T16:11:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.876908996Z", 2020-11-29T16:11:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YVkRWUTasfPQtpHA7NklgCE", 2020-11-29T16:11:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.881320_51.1", 2020-11-29T16:11:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14", 2020-11-29T16:11:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.881320871Z", 2020-11-29T16:11:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YVkRWUTasfPQtpHA7NklgCE", 2020-11-29T16:11:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.883170_52.1", 2020-11-29T16:11:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.054 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-14", 2020-11-29T16:11:14.054 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.883170592Z", 2020-11-29T16:11:14.054 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.054 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.054 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.055 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.055 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AsuYPfubiijDBKRDJK6qqjt", 2020-11-29T16:11:14.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.056 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.056 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.887114_53.1", 2020-11-29T16:11:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.057 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-14", 2020-11-29T16:11:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.887114146Z", 2020-11-29T16:11:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AsuYPfubiijDBKRDJK6qqjt", 2020-11-29T16:11:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.061 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.932301_54.1", 2020-11-29T16:11:14.061 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.061 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15", 2020-11-29T16:11:14.061 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.932301379Z", 2020-11-29T16:11:14.062 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.062 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.062 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.062 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.063 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.063 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_x_VdKNwXX4kJXxzVcmjduF6", 2020-11-29T16:11:14.063 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.064 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.064 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.065 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.936160_55.1", 2020-11-29T16:11:14.065 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.065 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15", 2020-11-29T16:11:14.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.936160887Z", 2020-11-29T16:11:14.066 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.066 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.066 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.066 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.067 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.067 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_x_VdKNwXX4kJXxzVcmjduF6", 2020-11-29T16:11:14.067 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.067 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.068 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.069 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.937891_56.1", 2020-11-29T16:11:14.069 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.069 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-15", 2020-11-29T16:11:14.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.937891458Z", 2020-11-29T16:11:14.069 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.070 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.070 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.070 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.071 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.071 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_QoRko9LJwKUXdid0giAhASQ", 2020-11-29T16:11:14.071 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.071 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.072 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.073 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665612.941872_57.1", 2020-11-29T16:11:14.073 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.073 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-15", 2020-11-29T16:11:14.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:00:12.941872760Z", 2020-11-29T16:11:14.073 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.074 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.074 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.074 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.074 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.075 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_QoRko9LJwKUXdid0giAhASQ", 2020-11-29T16:11:14.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.075 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.076 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.077 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.057028_59.1", 2020-11-29T16:11:14.077 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.077 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:14.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.057028183Z", 2020-11-29T16:11:14.077 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.078 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.080 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.061373_60.1", 2020-11-29T16:11:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.061373004Z", 2020-11-29T16:11:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.081 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.082 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.063911_61.1", 2020-11-29T16:11:14.084 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16", 2020-11-29T16:11:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.063911936Z", 2020-11-29T16:11:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.085 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qWQvPnGcRTlfP8YPDDWPU7j", 2020-11-29T16:11:14.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.069446_62.1", 2020-11-29T16:11:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.088 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16", 2020-11-29T16:11:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.069446267Z", 2020-11-29T16:11:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.089 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qWQvPnGcRTlfP8YPDDWPU7j", 2020-11-29T16:11:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.091 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.091 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.092 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.097339_63.1", 2020-11-29T16:11:14.092 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.092 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:14.092 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.097339684Z", 2020-11-29T16:11:14.093 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.093 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.093 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.094 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.094 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:14.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.095 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.096 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.102188_64.1", 2020-11-29T16:11:14.096 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.096 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:14.096 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.102188782Z", 2020-11-29T16:11:14.097 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.097 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.097 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.097 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.098 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.098 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:14.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.098 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.099 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.100 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.104862_65.1", 2020-11-29T16:11:14.100 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.100 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17", 2020-11-29T16:11:14.100 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.104862905Z", 2020-11-29T16:11:14.100 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.101 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.101 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.101 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.101 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.102 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3YDKF2LZHEWfgfUO2EvjsgL", 2020-11-29T16:11:14.102 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.102 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.102 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.103 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.103 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.103 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.103 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.104 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.108626_66.1", 2020-11-29T16:11:14.104 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.104 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17", 2020-11-29T16:11:14.104 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.108626431Z", 2020-11-29T16:11:14.104 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.105 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.105 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.105 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.105 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.106 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.106 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3YDKF2LZHEWfgfUO2EvjsgL", 2020-11-29T16:11:14.106 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.106 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.107 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.107 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.108 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.150629_67.1", 2020-11-29T16:11:14.108 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.108 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:14.108 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.150629298Z", 2020-11-29T16:11:14.108 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.109 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.109 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.109 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.110 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.111 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.158738_68.1", 2020-11-29T16:11:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.158738425Z", 2020-11-29T16:11:14.112 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.113 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.114 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.115 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.160845_69.1", 2020-11-29T16:11:14.116 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.116 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18", 2020-11-29T16:11:14.116 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.160845058Z", 2020-11-29T16:11:14.116 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.117 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.117 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.117 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.117 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.118 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5xSDI0jeDBdQUr4kb2mMHId", 2020-11-29T16:11:14.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.118 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.118 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.119 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.164732_70.1", 2020-11-29T16:11:14.120 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.120 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18", 2020-11-29T16:11:14.120 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.164732376Z", 2020-11-29T16:11:14.120 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.121 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.121 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.121 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.122 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5xSDI0jeDBdQUr4kb2mMHId", 2020-11-29T16:11:14.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.123 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.187080_71.1", 2020-11-29T16:11:14.124 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.124 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:14.124 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.187080319Z", 2020-11-29T16:11:14.124 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.124 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.125 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:14.125 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.125 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.126 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:14.126 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.126 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.126 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.127 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.127 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.127 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.195335_72.1", 2020-11-29T16:11:14.128 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.128 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16", 2020-11-29T16:11:14.128 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.195335470Z", 2020-11-29T16:11:14.128 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.128 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.129 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:14.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.129 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.129 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7H17euj1IqboxgktDzOKYuJ", 2020-11-29T16:11:14.130 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.130 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.130 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.131 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.131 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.131 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.131 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.203159_73.1", 2020-11-29T16:11:14.132 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.132 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:14.132 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.203159941Z", 2020-11-29T16:11:14.132 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.132 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.133 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:14.133 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.133 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.134 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:14.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.134 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.134 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.135 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.210320_74.1", 2020-11-29T16:11:14.136 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.136 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17", 2020-11-29T16:11:14.136 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.210320129Z", 2020-11-29T16:11:14.136 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.136 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.137 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:14.137 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.137 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.137 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KWCUWD45bPIJZ5IKnFnjaM1", 2020-11-29T16:11:14.138 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.138 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.138 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.139 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.139 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.221591_75.1", 2020-11-29T16:11:14.140 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.140 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:14.140 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.221591576Z", 2020-11-29T16:11:14.140 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.140 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.141 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:14.141 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.141 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.141 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:14.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.142 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.142 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.142 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.143 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.143 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665662.229493_76.1", 2020-11-29T16:11:14.143 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.144 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18", 2020-11-29T16:11:14.144 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:01:02.229493483Z", 2020-11-29T16:11:14.144 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.144 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.144 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:14.145 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.145 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.145 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_f5JGcUSWR0_DXOryRXcA4x3", 2020-11-29T16:11:14.145 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.146 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.146 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.147 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.147 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.147 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.967652_86.1", 2020-11-29T16:11:14.147 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.148 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:14.148 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.967652642Z", 2020-11-29T16:11:14.148 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.148 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.148 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.149 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.149 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.149 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:14.149 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.150 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.150 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.150 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.150 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.151 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.972279_87.1", 2020-11-29T16:11:14.151 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.151 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:14.152 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.972279097Z", 2020-11-29T16:11:14.152 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.152 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.152 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.153 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.153 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:14.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.154 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.154 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.155 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.155 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.974314_88.1", 2020-11-29T16:11:14.155 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.155 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19", 2020-11-29T16:11:14.156 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.974314737Z", 2020-11-29T16:11:14.156 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.156 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.156 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.156 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.157 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.157 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iGyiAtOMEjJAj1PjCXsPDZz", 2020-11-29T16:11:14.157 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.158 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.158 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.158 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.158 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.159 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.978564_89.1", 2020-11-29T16:11:14.159 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.159 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19", 2020-11-29T16:11:14.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.978564499Z", 2020-11-29T16:11:14.160 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.160 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.160 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.161 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.161 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iGyiAtOMEjJAj1PjCXsPDZz", 2020-11-29T16:11:14.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.162 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.162 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.163 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.984258_90.1", 2020-11-29T16:11:14.163 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.163 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:14.163 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.984258860Z", 2020-11-29T16:11:14.164 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.164 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.164 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:14.164 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.165 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.165 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:14.165 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.165 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665856.991097_91.1", 2020-11-29T16:11:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19", 2020-11-29T16:11:14.167 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:16.991097909Z", 2020-11-29T16:11:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oZxZm_Imstf5boZD4_wHnU1", 2020-11-29T16:11:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.169 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.170 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665997.215638_94.1", 2020-11-29T16:11:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.215638352Z", 2020-11-29T16:11:14.171 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.173 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665997.220413_95.1", 2020-11-29T16:11:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:14.175 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.220413426Z", 2020-11-29T16:11:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.177 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665997.222814_96.1", 2020-11-29T16:11:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20", 2020-11-29T16:11:14.179 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.222814425Z", 2020-11-29T16:11:14.180 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.180 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.180 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.180 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.181 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.181 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_caPUfpnBPovw7pnaLhfQKQu", 2020-11-29T16:11:14.181 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.182 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.182 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.183 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.183 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606665997.230793_97.1", 2020-11-29T16:11:14.183 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.183 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20", 2020-11-29T16:11:14.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:37.230793956Z", 2020-11-29T16:11:14.184 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.184 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.184 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.184 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.185 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.185 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_caPUfpnBPovw7pnaLhfQKQu", 2020-11-29T16:11:14.185 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.186 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.186 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.187 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666001.819331_99.1", 2020-11-29T16:11:14.187 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.187 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:14.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:41.819331435Z", 2020-11-29T16:11:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666001.830749_100.1", 2020-11-29T16:11:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:06:41.830749663Z", 2020-11-29T16:11:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666061.809169_106.1", 2020-11-29T16:11:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:07:41.809169955Z", 2020-11-29T16:11:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666061.819824_107.1", 2020-11-29T16:11:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20", 2020-11-29T16:11:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:07:41.819824059Z", 2020-11-29T16:11:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vz1sw62cuQNoRNZICnOuKnz", 2020-11-29T16:11:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.202 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.202 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.732730_108.1", 2020-11-29T16:11:14.203 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.203 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:14.203 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.732730870Z", 2020-11-29T16:11:14.203 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.204 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.204 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.204 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.204 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.205 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.206 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.738936_109.1", 2020-11-29T16:11:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.738936839Z", 2020-11-29T16:11:14.207 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.208 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.209 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:14.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.209 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.209 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.210 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.740862_110.1", 2020-11-29T16:11:14.211 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.211 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21", 2020-11-29T16:11:14.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.740862527Z", 2020-11-29T16:11:14.211 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.212 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.212 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.212 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.213 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1LtU-sUHWZQfFCaHHia4XL6", 2020-11-29T16:11:14.213 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.213 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.213 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.214 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.214 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.214 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.745932_111.1", 2020-11-29T16:11:14.215 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.215 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21", 2020-11-29T16:11:14.215 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.745932979Z", 2020-11-29T16:11:14.215 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.216 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.216 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.216 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.216 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.217 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1LtU-sUHWZQfFCaHHia4XL6", 2020-11-29T16:11:14.217 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.217 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.217 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.218 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.218 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.218 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.771777_112.1", 2020-11-29T16:11:14.219 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.219 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:14.219 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.771777245Z", 2020-11-29T16:11:14.219 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.219 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.220 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.220 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.220 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.221 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:14.221 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.221 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.221 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.222 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.779214_113.1", 2020-11-29T16:11:14.223 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.223 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:14.223 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.779214800Z", 2020-11-29T16:11:14.223 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.224 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.224 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.224 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.225 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:14.225 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.225 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.225 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.226 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.781033_114.1", 2020-11-29T16:11:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22", 2020-11-29T16:11:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.781033305Z", 2020-11-29T16:11:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.227 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.228 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yAIfpIe3NidR6qhQ52Y6gQ9", 2020-11-29T16:11:14.229 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.229 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.229 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.230 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.786435_115.1", 2020-11-29T16:11:14.231 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.231 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22", 2020-11-29T16:11:14.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.786435846Z", 2020-11-29T16:11:14.231 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.231 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.232 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yAIfpIe3NidR6qhQ52Y6gQ9", 2020-11-29T16:11:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.234 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.234 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.234 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.833583_116.1", 2020-11-29T16:11:14.234 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.235 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:14.235 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.833583970Z", 2020-11-29T16:11:14.235 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.235 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.235 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.236 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.236 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.236 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.237 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.238 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.838249_117.1", 2020-11-29T16:11:14.238 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.239 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:14.239 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.838249602Z", 2020-11-29T16:11:14.239 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.239 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.239 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.240 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.240 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:14.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.840897_118.1", 2020-11-29T16:11:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23", 2020-11-29T16:11:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.840897485Z", 2020-11-29T16:11:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yUDBs52ibfejiizLQQANf-x", 2020-11-29T16:11:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.846339_119.1", 2020-11-29T16:11:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23", 2020-11-29T16:11:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.846339269Z", 2020-11-29T16:11:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yUDBs52ibfejiizLQQANf-x", 2020-11-29T16:11:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.250 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.250 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.869230_120.1", 2020-11-29T16:11:14.250 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.250 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:14.250 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.869230022Z", 2020-11-29T16:11:14.251 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.251 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.251 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:14.251 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.252 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.252 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:14.252 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.252 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.253 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.254 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.254 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.875173_121.1", 2020-11-29T16:11:14.254 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.254 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21", 2020-11-29T16:11:14.254 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.875173699Z", 2020-11-29T16:11:14.255 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.255 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.255 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:14.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.256 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.256 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_G58fvPPefeSNYaX26cvah6v", 2020-11-29T16:11:14.256 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.256 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.257 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.258 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.879784_122.1", 2020-11-29T16:11:14.258 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.258 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:14.258 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.879784537Z", 2020-11-29T16:11:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.260 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.260 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:14.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.260 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.262 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.886953_123.1", 2020-11-29T16:11:14.262 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.262 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22", 2020-11-29T16:11:14.262 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.886953932Z", 2020-11-29T16:11:14.262 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.263 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.263 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:14.263 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.263 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.264 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AaTayb5Y0UTlDvT8mVUu0QS", 2020-11-29T16:11:14.264 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.264 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.264 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.265 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.265 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.901158_124.1", 2020-11-29T16:11:14.266 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.266 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:14.266 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.901158656Z", 2020-11-29T16:11:14.266 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.267 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.267 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:14.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.267 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.268 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:14.268 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.268 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.268 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.269 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.269 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.269 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666101.915755_125.1", 2020-11-29T16:11:14.270 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.270 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23", 2020-11-29T16:11:14.270 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:21.915755723Z", 2020-11-29T16:11:14.270 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Opm7sORgIkSGqo7XLHtMwxa", 2020-11-29T16:11:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.456443_126.1", 2020-11-29T16:11:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.456443611Z", 2020-11-29T16:11:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.460387_127.1", 2020-11-29T16:11:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.460387996Z", 2020-11-29T16:11:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.279 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.279 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.279 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.280 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.280 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.280 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.280 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.281 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.462193_128.1", 2020-11-29T16:11:14.282 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.282 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24", 2020-11-29T16:11:14.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.462193185Z", 2020-11-29T16:11:14.282 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.283 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.283 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.283 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_8rFe9Ex-HuV_dw_B2ecHBhB", 2020-11-29T16:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.285 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.467714_129.1", 2020-11-29T16:11:14.286 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.286 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24", 2020-11-29T16:11:14.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.467714061Z", 2020-11-29T16:11:14.286 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.286 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.287 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.287 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.287 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.288 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_8rFe9Ex-HuV_dw_B2ecHBhB", 2020-11-29T16:11:14.288 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.289 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.490669_130.1", 2020-11-29T16:11:14.290 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.290 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:14.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.490669098Z", 2020-11-29T16:11:14.290 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.290 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.291 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.291 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.291 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.291 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.292 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.292 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.292 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.293 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.496194_131.1", 2020-11-29T16:11:14.294 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.294 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:14.294 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.496194848Z", 2020-11-29T16:11:14.294 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.294 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.295 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.295 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.295 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.295 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.296 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.296 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.296 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.297 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.297 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.498339_132.1", 2020-11-29T16:11:14.298 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.298 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25", 2020-11-29T16:11:14.298 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.498339597Z", 2020-11-29T16:11:14.298 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.298 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.299 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.299 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.299 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s63DKGRwg1GnITE9VweheW4", 2020-11-29T16:11:14.300 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.300 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.300 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.300 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.301 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.301 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.301 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.502532_133.1", 2020-11-29T16:11:14.302 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.302 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25", 2020-11-29T16:11:14.302 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.502532461Z", 2020-11-29T16:11:14.302 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.302 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.303 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.303 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.303 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s63DKGRwg1GnITE9VweheW4", 2020-11-29T16:11:14.304 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.304 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.304 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.304 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.305 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.305 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.557238_134.1", 2020-11-29T16:11:14.305 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.306 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:14.306 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.557238717Z", 2020-11-29T16:11:14.306 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.306 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.307 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.307 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.307 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.307 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.308 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.308 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.308 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.309 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.562930_135.1", 2020-11-29T16:11:14.309 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.310 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:14.310 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.562930060Z", 2020-11-29T16:11:14.310 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.310 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.310 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.311 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.311 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.311 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.312 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.312 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.312 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.312 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.313 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.565930_136.1", 2020-11-29T16:11:14.313 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.314 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26", 2020-11-29T16:11:14.314 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.565930147Z", 2020-11-29T16:11:14.314 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.314 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.314 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.315 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.315 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.315 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HEKlnFCl4gKqxl9FXC6_Slr", 2020-11-29T16:11:14.315 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.316 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.316 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.316 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.316 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.317 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.317 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.574718_137.1", 2020-11-29T16:11:14.317 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.318 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26", 2020-11-29T16:11:14.318 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.574718238Z", 2020-11-29T16:11:14.318 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.318 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.318 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.319 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.319 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.319 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HEKlnFCl4gKqxl9FXC6_Slr", 2020-11-29T16:11:14.319 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.320 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.594772_138.1", 2020-11-29T16:11:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.321 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.594772332Z", 2020-11-29T16:11:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.322 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.323 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.599622_139.1", 2020-11-29T16:11:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.325 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.599622375Z", 2020-11-29T16:11:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.326 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.603089_140.1", 2020-11-29T16:11:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.329 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.603089353Z", 2020-11-29T16:11:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:14.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.333 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.612024_141.1", 2020-11-29T16:11:14.333 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.333 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24", 2020-11-29T16:11:14.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.612024636Z", 2020-11-29T16:11:14.334 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.334 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.334 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.335 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.335 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wSD5w84M2-VeVMLTDey2Sqq", 2020-11-29T16:11:14.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:14.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.336 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.336 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.337 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.616582_142.1", 2020-11-29T16:11:14.337 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.337 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:14.337 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.616582099Z", 2020-11-29T16:11:14.338 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.338 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.338 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.338 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.339 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.339 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.340 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.340 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.341 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.624065_143.1", 2020-11-29T16:11:14.341 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.341 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:14.341 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.624065742Z", 2020-11-29T16:11:14.342 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.342 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.342 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.342 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.343 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.343 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.343 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.343 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.344 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.345 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.633769_144.1", 2020-11-29T16:11:14.345 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.345 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:14.345 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.633769558Z", 2020-11-29T16:11:14.346 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.346 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.346 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.346 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.347 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.347 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.347 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:14.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.347 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.348 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.348 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.349 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.641875_145.1", 2020-11-29T16:11:14.349 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.349 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25", 2020-11-29T16:11:14.349 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.641875589Z", 2020-11-29T16:11:14.349 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.350 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.350 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.350 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.350 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.351 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YwQjhnncQpW4cA7viswxZmK", 2020-11-29T16:11:14.351 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:14.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.351 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.655938_146.1", 2020-11-29T16:11:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.655938546Z", 2020-11-29T16:11:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.665066_147.1", 2020-11-29T16:11:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.665066109Z", 2020-11-29T16:11:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.677379_148.1", 2020-11-29T16:11:14.361 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.361 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:14.361 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.677379068Z", 2020-11-29T16:11:14.361 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.361 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.362 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.362 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.362 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.363 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.363 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:14.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.363 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.363 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.364 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666112.684234_149.1", 2020-11-29T16:11:14.365 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.365 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26", 2020-11-29T16:11:14.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:32.684234315Z", 2020-11-29T16:11:14.365 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.365 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.366 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.366 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.366 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.366 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FZJCM6UEQEKdLRkkay7oe1E", 2020-11-29T16:11:14.367 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-11-29T16:11:14.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.367 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.367 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.368 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.338993_154.1", 2020-11-29T16:11:14.369 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.369 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:14.369 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.338993057Z", 2020-11-29T16:11:14.369 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.369 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.370 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.370 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.370 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.371 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:14.371 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.371 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.371 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.372 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.346126_155.1", 2020-11-29T16:11:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.346126383Z", 2020-11-29T16:11:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.373 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.374 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.374 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.374 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.374 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:14.375 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.375 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.375 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.376 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.348222_156.1", 2020-11-29T16:11:14.377 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.377 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27", 2020-11-29T16:11:14.377 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.348222396Z", 2020-11-29T16:11:14.377 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.377 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.378 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.378 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.378 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.378 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Wc4auwc30IepX657G2eZM-u", 2020-11-29T16:11:14.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.379 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.379 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.380 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.353951_157.1", 2020-11-29T16:11:14.381 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.381 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27", 2020-11-29T16:11:14.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.353951918Z", 2020-11-29T16:11:14.381 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.381 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.382 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.382 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.382 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Wc4auwc30IepX657G2eZM-u", 2020-11-29T16:11:14.383 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.383 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.383 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.384 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.390106_158.1", 2020-11-29T16:11:14.385 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.385 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:14.385 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.390106865Z", 2020-11-29T16:11:14.385 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.385 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.386 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.386 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.386 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.386 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:14.387 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.387 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.387 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.388 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.397881_159.1", 2020-11-29T16:11:14.389 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.389 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:14.389 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.397881409Z", 2020-11-29T16:11:14.389 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.389 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.390 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.390 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.390 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.391 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:14.391 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.391 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.391 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.392 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.400119_160.1", 2020-11-29T16:11:14.393 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.393 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28", 2020-11-29T16:11:14.393 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.400119169Z", 2020-11-29T16:11:14.393 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.393 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.394 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.394 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.394 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.395 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vorB3p52YuzxIbSF-GHscIz", 2020-11-29T16:11:14.395 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.395 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.395 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.396 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.404155_161.1", 2020-11-29T16:11:14.397 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.397 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28", 2020-11-29T16:11:14.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.404155690Z", 2020-11-29T16:11:14.397 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.397 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.398 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.398 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.398 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.399 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_vorB3p52YuzxIbSF-GHscIz", 2020-11-29T16:11:14.401 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.402 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.402 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.403 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.461726_162.1", 2020-11-29T16:11:14.404 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.404 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:14.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.461726137Z", 2020-11-29T16:11:14.404 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.405 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.405 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.405 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.405 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.406 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:14.406 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.406 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.407 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.407 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.408 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.466239_163.1", 2020-11-29T16:11:14.408 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.408 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:14.408 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.466239727Z", 2020-11-29T16:11:14.409 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.409 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.409 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.409 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.410 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.410 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:14.410 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.411 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.411 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.412 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.471274_164.1", 2020-11-29T16:11:14.412 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.412 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29", 2020-11-29T16:11:14.412 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.471274792Z", 2020-11-29T16:11:14.413 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.413 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.413 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.413 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.414 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.414 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tW9vOLbEcF0UPbuJYTP9PYC", 2020-11-29T16:11:14.414 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.415 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.415 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.416 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.475990_165.1", 2020-11-29T16:11:14.416 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.416 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29", 2020-11-29T16:11:14.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.475990005Z", 2020-11-29T16:11:14.417 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.417 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.417 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.418 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.418 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tW9vOLbEcF0UPbuJYTP9PYC", 2020-11-29T16:11:14.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.419 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.419 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.420 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.420 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.500752_166.1", 2020-11-29T16:11:14.420 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.421 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:14.421 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.500752451Z", 2020-11-29T16:11:14.421 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.421 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.421 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:14.422 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.422 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.422 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:14.423 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.423 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.423 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.424 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.506749_167.1", 2020-11-29T16:11:14.425 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.425 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-27:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27", 2020-11-29T16:11:14.425 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.506749250Z", 2020-11-29T16:11:14.425 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.426 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.426 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:14.426 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.426 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.427 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1TUVOsPe8AycEcIjwOEyOnj", 2020-11-29T16:11:14.427 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.427 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.427 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.428 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.428 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.428 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.428 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.561039_168.1", 2020-11-29T16:11:14.429 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.429 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:14.429 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.561039789Z", 2020-11-29T16:11:14.429 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.430 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.430 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:14.430 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.430 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.431 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:14.431 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.431 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.432 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.433 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.568104_169.1", 2020-11-29T16:11:14.433 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.433 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-28:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28", 2020-11-29T16:11:14.433 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.568104840Z", 2020-11-29T16:11:14.433 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.434 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.434 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:14.434 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.435 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.435 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VYmfYnHMOqfIHvrXzAZTwO9", 2020-11-29T16:11:14.435 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.435 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.436 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.437 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.624595_170.1", 2020-11-29T16:11:14.437 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.437 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:14.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.624595921Z", 2020-11-29T16:11:14.438 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.438 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.438 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:14.438 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.439 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.439 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:14.439 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.631524_171.1", 2020-11-29T16:11:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-29:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29", 2020-11-29T16:11:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.631524395Z", 2020-11-29T16:11:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rzIfOPwAIj90bqR1VjdkgHS", 2020-11-29T16:11:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.709256_172.1", 2020-11-29T16:11:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.709256940Z", 2020-11-29T16:11:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.446 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:14.447 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.713773_173.1", 2020-11-29T16:11:14.449 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.713773728Z", 2020-11-29T16:11:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.450 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:14.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.453 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.453 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.718379_174.1", 2020-11-29T16:11:14.453 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.454 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30", 2020-11-29T16:11:14.454 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.718379824Z", 2020-11-29T16:11:14.454 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.454 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.454 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.455 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.455 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.455 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h_WBzhTZe664PHorKv6_Am0", 2020-11-29T16:11:14.455 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.456 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.456 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.456 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.457 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.457 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.457 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.723686_175.1", 2020-11-29T16:11:14.457 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.458 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30", 2020-11-29T16:11:14.458 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.723686356Z", 2020-11-29T16:11:14.458 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.458 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.458 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.459 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.459 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h_WBzhTZe664PHorKv6_Am0", 2020-11-29T16:11:14.460 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.460 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.460 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.460 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.461 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.752571_176.1", 2020-11-29T16:11:14.461 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.462 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:14.462 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.752571930Z", 2020-11-29T16:11:14.462 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.462 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.463 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.463 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.463 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.463 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:14.464 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.464 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.464 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.465 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.465 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.756904_177.1", 2020-11-29T16:11:14.466 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.466 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:14.466 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.756904681Z", 2020-11-29T16:11:14.466 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.466 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.467 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.467 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.467 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.468 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:14.468 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.468 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.468 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.469 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.469 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.469 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.759821_178.1", 2020-11-29T16:11:14.470 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.470 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31", 2020-11-29T16:11:14.470 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.759821922Z", 2020-11-29T16:11:14.470 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.471 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.471 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.471 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.471 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.472 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_eKjXcKGmg5yyLHdihUsR2sW", 2020-11-29T16:11:14.472 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.472 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.473 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.474 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.764291_179.1", 2020-11-29T16:11:14.474 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.474 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31", 2020-11-29T16:11:14.474 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.764291393Z", 2020-11-29T16:11:14.474 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.475 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.475 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.475 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.476 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.476 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_eKjXcKGmg5yyLHdihUsR2sW", 2020-11-29T16:11:14.476 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.476 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.477 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.478 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.825137_180.1", 2020-11-29T16:11:14.478 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.478 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:14.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.825137534Z", 2020-11-29T16:11:14.478 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.479 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.479 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.479 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.480 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.480 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:14.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.480 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.482 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.829684_181.1", 2020-11-29T16:11:14.482 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.482 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:14.482 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.829684939Z", 2020-11-29T16:11:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.484 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.484 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:14.484 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.485 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.485 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.486 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.831781_182.1", 2020-11-29T16:11:14.486 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.486 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32", 2020-11-29T16:11:14.486 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.831781072Z", 2020-11-29T16:11:14.487 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.487 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.487 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.487 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.488 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.488 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dhvj1N7nCxk5qukpVfy3IjW", 2020-11-29T16:11:14.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.489 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.489 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.490 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.835966_183.1", 2020-11-29T16:11:14.490 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.490 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32", 2020-11-29T16:11:14.491 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.835966939Z", 2020-11-29T16:11:14.491 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.491 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.491 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.491 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.492 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.492 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dhvj1N7nCxk5qukpVfy3IjW", 2020-11-29T16:11:14.492 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.493 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.493 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.494 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.860087_184.1", 2020-11-29T16:11:14.494 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.494 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:14.495 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.860087065Z", 2020-11-29T16:11:14.495 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.495 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.495 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:14.495 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.496 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.496 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:14.496 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.497 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.497 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.497 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.497 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.498 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.865870_185.1", 2020-11-29T16:11:14.498 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.499 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-30:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30", 2020-11-29T16:11:14.499 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.865870483Z", 2020-11-29T16:11:14.499 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.499 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.499 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:14.500 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.500 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.500 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ocgDDLA4AQV7yY9hZcXoS85", 2020-11-29T16:11:14.501 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.501 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.501 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.501 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.502 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.914709_186.1", 2020-11-29T16:11:14.502 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.503 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:14.503 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.914709667Z", 2020-11-29T16:11:14.503 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.503 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.504 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:14.504 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.504 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.504 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.925154_187.1", 2020-11-29T16:11:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-31:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31", 2020-11-29T16:11:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.925154339Z", 2020-11-29T16:11:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_paMeplB1XjaHyC6MHIShz4p", 2020-11-29T16:11:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.991321_188.1", 2020-11-29T16:11:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.991321647Z", 2020-11-29T16:11:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.515 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666122.999983_189.1", 2020-11-29T16:11:14.515 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.515 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-32:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32", 2020-11-29T16:11:14.515 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:42.999983004Z", 2020-11-29T16:11:14.515 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.516 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.516 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:14.516 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.516 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.517 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5eaNlQCx9Vze9bUJPVEAXis", 2020-11-29T16:11:14.517 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.517 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.517 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.518 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.518 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.519 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.076605_190.1", 2020-11-29T16:11:14.519 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.519 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:14.519 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.076605830Z", 2020-11-29T16:11:14.520 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.520 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.520 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.520 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.522 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.522 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.523 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.081952_191.1", 2020-11-29T16:11:14.523 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.523 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:14.524 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.081952008Z", 2020-11-29T16:11:14.524 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.524 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.524 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.524 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.525 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.525 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:14.525 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.526 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.526 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.526 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.526 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.527 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.527 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.527 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.083820_192.1", 2020-11-29T16:11:14.527 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.527 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33", 2020-11-29T16:11:14.528 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.083820333Z", 2020-11-29T16:11:14.528 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.528 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.528 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.528 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.529 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.529 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_aScNky5UrTuH12xWR79F83e", 2020-11-29T16:11:14.529 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.530 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.530 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.530 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.530 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.531 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.531 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.531 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.088982_193.1", 2020-11-29T16:11:14.531 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.531 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33", 2020-11-29T16:11:14.532 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.088982559Z", 2020-11-29T16:11:14.532 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.532 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.532 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.533 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.533 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_aScNky5UrTuH12xWR79F83e", 2020-11-29T16:11:14.533 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.534 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.534 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.535 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.121086_194.1", 2020-11-29T16:11:14.535 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.535 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:14.536 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.121086133Z", 2020-11-29T16:11:14.536 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.536 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.536 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.536 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.537 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.537 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:14.537 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.539 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.133205_195.1", 2020-11-29T16:11:14.539 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.539 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:14.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.133205341Z", 2020-11-29T16:11:14.540 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.540 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.540 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.541 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.541 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:14.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.542 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.542 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.543 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.138006_196.1", 2020-11-29T16:11:14.543 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34", 2020-11-29T16:11:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.138006291Z", 2020-11-29T16:11:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.544 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_9r_K5jlLAncl40ETfoVeJjK", 2020-11-29T16:11:14.545 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.546 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.546 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.547 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.153529_197.1", 2020-11-29T16:11:14.547 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.548 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34", 2020-11-29T16:11:14.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.153529708Z", 2020-11-29T16:11:14.548 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.548 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.548 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.549 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.549 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.549 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_9r_K5jlLAncl40ETfoVeJjK", 2020-11-29T16:11:14.549 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.550 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.550 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.551 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.551 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.225476_198.1", 2020-11-29T16:11:14.551 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.552 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:14.552 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.225476965Z", 2020-11-29T16:11:14.552 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.552 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.553 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:14.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.554 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.554 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.555 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.230332_199.1", 2020-11-29T16:11:14.555 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.556 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:14.556 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.230332153Z", 2020-11-29T16:11:14.556 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.556 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.557 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.557 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.557 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.557 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.232895_200.1", 2020-11-29T16:11:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35", 2020-11-29T16:11:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.232895662Z", 2020-11-29T16:11:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.561 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.561 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.561 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.561 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BauiNjhDLe4JD_SZUYzW5BH", 2020-11-29T16:11:14.562 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.562 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.562 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.562 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.563 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.563 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.237930_201.1", 2020-11-29T16:11:14.563 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.564 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35", 2020-11-29T16:11:14.564 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.237930233Z", 2020-11-29T16:11:14.564 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.564 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.564 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.565 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.565 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.565 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BauiNjhDLe4JD_SZUYzW5BH", 2020-11-29T16:11:14.565 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.566 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.566 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.566 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.567 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.567 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.261144_202.1", 2020-11-29T16:11:14.567 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.568 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:14.568 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.261144444Z", 2020-11-29T16:11:14.568 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.568 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.569 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:14.569 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.569 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.569 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:14.570 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.570 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.570 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.571 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666123.266462_203.1", 2020-11-29T16:11:14.572 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.572 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-33:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33", 2020-11-29T16:11:14.572 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:43.266462524Z", 2020-11-29T16:11:14.572 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.573 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.573 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:14.573 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.573 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.574 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bwVO2HM_dy3-T-VnpmGuwsC", 2020-11-29T16:11:14.574 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.574 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.574 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.575 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666124.313715_204.1", 2020-11-29T16:11:14.576 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.576 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:14.576 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:44.313715914Z", 2020-11-29T16:11:14.576 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.576 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.577 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:14.577 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.577 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.578 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:14.578 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.578 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.578 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.579 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.579 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.579 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666124.327834_205.1", 2020-11-29T16:11:14.580 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.580 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-34:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34", 2020-11-29T16:11:14.580 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:44.327834836Z", 2020-11-29T16:11:14.580 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.580 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.581 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:14.581 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.581 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.581 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3s5ozm4W_fennng-opnJNz2", 2020-11-29T16:11:14.582 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.582 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.582 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.583 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.389734_206.1", 2020-11-29T16:11:14.584 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.584 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:14.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.389734131Z", 2020-11-29T16:11:14.584 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.585 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.585 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:14.585 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.585 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.586 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:14.586 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.586 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.586 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.588 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.399959_207.1", 2020-11-29T16:11:14.588 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.588 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-35:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35", 2020-11-29T16:11:14.588 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.399959602Z", 2020-11-29T16:11:14.588 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.589 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.589 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:14.589 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.589 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.590 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_B2Mo1d_taqMWrcddYBraC6G", 2020-11-29T16:11:14.590 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-11-29T16:11:14.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.590 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.591 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.592 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.480704_208.1", 2020-11-29T16:11:14.592 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.592 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36", 2020-11-29T16:11:14.592 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.480704401Z", 2020-11-29T16:11:14.592 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.593 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.593 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.593 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.593 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.594 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_UJxOrACWaVFo5tZbkaJd5aB", 2020-11-29T16:11:14.594 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.594 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.594 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.595 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.595 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.595 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.485309_209.1", 2020-11-29T16:11:14.596 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.596 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36", 2020-11-29T16:11:14.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.485309628Z", 2020-11-29T16:11:14.596 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.596 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.597 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.597 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.597 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.597 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_UJxOrACWaVFo5tZbkaJd5aB", 2020-11-29T16:11:14.598 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.598 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.598 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.599 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.487975_210.1", 2020-11-29T16:11:14.600 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.600 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-36", 2020-11-29T16:11:14.600 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.487975960Z", 2020-11-29T16:11:14.600 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.601 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.601 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.601 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.601 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.601 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VZJxPfFqHBrvBon1FcGTxPs", 2020-11-29T16:11:14.602 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.602 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.602 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.603 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.603 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.603 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.491713_211.1", 2020-11-29T16:11:14.604 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.604 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-36", 2020-11-29T16:11:14.604 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.491713665Z", 2020-11-29T16:11:14.604 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.604 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.605 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.605 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.605 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.605 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_VZJxPfFqHBrvBon1FcGTxPs", 2020-11-29T16:11:14.605 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.606 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.606 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.607 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.525177_212.1", 2020-11-29T16:11:14.607 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.608 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37", 2020-11-29T16:11:14.608 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.525177090Z", 2020-11-29T16:11:14.608 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.608 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.608 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_syk3UA8fSzEvnh7yGHxS7iU", 2020-11-29T16:11:14.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.611 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.534079_213.1", 2020-11-29T16:11:14.611 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.612 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37", 2020-11-29T16:11:14.612 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.534079771Z", 2020-11-29T16:11:14.612 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.612 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.612 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.613 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.613 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.613 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_syk3UA8fSzEvnh7yGHxS7iU", 2020-11-29T16:11:14.613 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.614 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.614 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.615 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.540941_214.1", 2020-11-29T16:11:14.615 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.616 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-37", 2020-11-29T16:11:14.616 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.540941786Z", 2020-11-29T16:11:14.616 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.616 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.616 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.617 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.617 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_LPLM1eQYX53fxff8emhXqjO", 2020-11-29T16:11:14.618 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.618 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.618 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.619 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.619 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.547825_215.1", 2020-11-29T16:11:14.619 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.620 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-37", 2020-11-29T16:11:14.620 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.547825214Z", 2020-11-29T16:11:14.620 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.620 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_LPLM1eQYX53fxff8emhXqjO", 2020-11-29T16:11:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.590883_216.1", 2020-11-29T16:11:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38", 2020-11-29T16:11:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.590883464Z", 2020-11-29T16:11:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.625 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.625 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.625 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_sluiDwZnW0jlavok--xXUQM", 2020-11-29T16:11:14.625 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.595783_217.1", 2020-11-29T16:11:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38", 2020-11-29T16:11:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.595783563Z", 2020-11-29T16:11:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_sluiDwZnW0jlavok--xXUQM", 2020-11-29T16:11:14.630 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.630 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.630 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.631 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.631 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.631 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.597926_218.1", 2020-11-29T16:11:14.632 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.632 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-38", 2020-11-29T16:11:14.632 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.597926148Z", 2020-11-29T16:11:14.632 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.632 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.633 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.633 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.633 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.633 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s5CCY0lZsRXaG97sebP-L5L", 2020-11-29T16:11:14.634 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.634 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.634 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.635 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666125.602582_219.1", 2020-11-29T16:11:14.636 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.636 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-38", 2020-11-29T16:11:14.636 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:45.602582602Z", 2020-11-29T16:11:14.636 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.636 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s5CCY0lZsRXaG97sebP-L5L", 2020-11-29T16:11:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.333843_222.1", 2020-11-29T16:11:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-39:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39", 2020-11-29T16:11:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.333843820Z", 2020-11-29T16:11:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5H0pP9yCjtoMl1W4TvUgcb8", 2020-11-29T16:11:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.338434_223.1", 2020-11-29T16:11:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-39:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39", 2020-11-29T16:11:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.338434096Z", 2020-11-29T16:11:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5H0pP9yCjtoMl1W4TvUgcb8", 2020-11-29T16:11:14.646 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.646 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.646 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.647 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.340224_224.1", 2020-11-29T16:11:14.647 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.648 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-39", 2020-11-29T16:11:14.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.340224571Z", 2020-11-29T16:11:14.648 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.648 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.649 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.649 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.649 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.649 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Pp8cXl7djO1gxLhMq2AZXJp", 2020-11-29T16:11:14.650 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.650 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.650 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.651 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.343737_225.1", 2020-11-29T16:11:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-39", 2020-11-29T16:11:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.343737416Z", 2020-11-29T16:11:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.652 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.653 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Pp8cXl7djO1gxLhMq2AZXJp", 2020-11-29T16:11:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.654 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.655 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.368575_226.1", 2020-11-29T16:11:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-40:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40", 2020-11-29T16:11:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.368575114Z", 2020-11-29T16:11:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.656 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.657 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jgn-ZwFyVxv-tmED-Ggd4Hw", 2020-11-29T16:11:14.658 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.658 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.658 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.659 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.374510_227.1", 2020-11-29T16:11:14.659 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.660 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-40:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40", 2020-11-29T16:11:14.660 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.374510224Z", 2020-11-29T16:11:14.660 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.660 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.661 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.661 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.661 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.661 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jgn-ZwFyVxv-tmED-Ggd4Hw", 2020-11-29T16:11:14.662 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.662 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.662 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.663 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.376835_228.1", 2020-11-29T16:11:14.664 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.664 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-40", 2020-11-29T16:11:14.664 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.376835132Z", 2020-11-29T16:11:14.664 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.664 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.665 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.665 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.665 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.665 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qrn1nKXKg0aZIU2cqKpKgju", 2020-11-29T16:11:14.666 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.666 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.666 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.667 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.380733_229.1", 2020-11-29T16:11:14.667 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.668 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-40", 2020-11-29T16:11:14.668 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.380733954Z", 2020-11-29T16:11:14.668 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.668 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.668 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.669 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.669 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.669 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qrn1nKXKg0aZIU2cqKpKgju", 2020-11-29T16:11:14.669 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.670 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.670 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.670 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.671 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.430813_231.1", 2020-11-29T16:11:14.671 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.672 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-41:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41", 2020-11-29T16:11:14.672 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.430813301Z", 2020-11-29T16:11:14.672 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.672 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.672 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.673 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.673 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.673 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_upM-2-zq2ujNZABi5Lo2I6j", 2020-11-29T16:11:14.674 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.674 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.674 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.674 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.675 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.440999_232.1", 2020-11-29T16:11:14.675 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.676 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-41:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41", 2020-11-29T16:11:14.676 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.440999402Z", 2020-11-29T16:11:14.676 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.676 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.677 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.677 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.677 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_upM-2-zq2ujNZABi5Lo2I6j", 2020-11-29T16:11:14.678 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.678 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.678 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.678 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.679 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.444219_234.1", 2020-11-29T16:11:14.679 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.680 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-41", 2020-11-29T16:11:14.680 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.444219056Z", 2020-11-29T16:11:14.680 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.680 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iwxqSbv55pvq-WwQMf1oR3e", 2020-11-29T16:11:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666127.452875_235.1", 2020-11-29T16:11:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-41", 2020-11-29T16:11:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.452875466Z", 2020-11-29T16:11:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:14.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:14.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.685 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:14.685 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iwxqSbv55pvq-WwQMf1oR3e", 2020-11-29T16:11:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:14.687 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:14.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:14.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:14.690 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f053a404-d473-420f-9a01-38eedc856256 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:14.729 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:14.730 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:14.969 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:14.969 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:14.970 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.848433_141.1", 2020-11-29T16:11:14.970 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.970 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-42:3c165286-3738-4e0f-b399-acd859302998.5116.1", 2020-11-29T16:11:14.970 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.848433092Z", 2020-11-29T16:11:14.971 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:14.971 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:14.971 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:14.971 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:14.971 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.972 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:14.972 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_uJHlSItgASeLHf4vV6myM3g", 2020-11-29T16:11:14.972 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:14.972 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.973 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:14.973 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:14.973 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.973 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.973 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.974 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:14.974 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.852884_142.1", 2020-11-29T16:11:14.974 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.974 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-42:3c165286-3738-4e0f-b399-acd859302998.5116.1", 2020-11-29T16:11:14.975 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.852884445Z", 2020-11-29T16:11:14.975 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:14.975 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:14.975 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:14.975 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:14.976 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.976 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:14.976 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_uJHlSItgASeLHf4vV6myM3g", 2020-11-29T16:11:14.976 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:14.976 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.977 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:14.977 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:14.977 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.977 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.978 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.978 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:14.978 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.854739_143.1", 2020-11-29T16:11:14.978 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:14.978 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-42", 2020-11-29T16:11:14.979 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.854739278Z", 2020-11-29T16:11:14.979 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:14.979 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:14.979 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:14.979 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:14.980 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.980 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:14.980 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1OxpMJPB0AzK0o5_DAd_Re8", 2020-11-29T16:11:14.980 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:14.981 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.981 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:14.981 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:14.981 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.981 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.982 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.982 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:14.982 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.862141_144.1", 2020-11-29T16:11:14.982 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:14.982 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-42", 2020-11-29T16:11:14.983 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.862141657Z", 2020-11-29T16:11:14.983 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:14.983 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:14.983 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:14.983 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:14.984 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.984 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:14.984 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1OxpMJPB0AzK0o5_DAd_Re8", 2020-11-29T16:11:14.984 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:14.984 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.985 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:14.985 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:14.985 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.985 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.986 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.986 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:14.986 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.912191_145.1", 2020-11-29T16:11:14.986 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.986 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-43:3c165286-3738-4e0f-b399-acd859302998.5116.2", 2020-11-29T16:11:14.987 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.912191123Z", 2020-11-29T16:11:14.987 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:14.987 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:14.987 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:14.987 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:14.988 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.988 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:14.988 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tIFKXdQQP_3E_YrQvx0Z5eu", 2020-11-29T16:11:14.988 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:14.988 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.989 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:14.989 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:14.989 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.989 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.990 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.990 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:14.990 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.919404_146.1", 2020-11-29T16:11:14.990 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:14.990 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-43:3c165286-3738-4e0f-b399-acd859302998.5116.2", 2020-11-29T16:11:14.991 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.919404670Z", 2020-11-29T16:11:14.991 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:14.991 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:14.991 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:14.991 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:14.992 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.992 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:14.992 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tIFKXdQQP_3E_YrQvx0Z5eu", 2020-11-29T16:11:14.992 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:14.992 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.993 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:14.993 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:14.993 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.993 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.994 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.994 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:14.994 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.921398_147.1", 2020-11-29T16:11:14.994 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:14.994 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-43", 2020-11-29T16:11:14.995 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.921398751Z", 2020-11-29T16:11:14.995 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:14.995 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:14.995 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:14.995 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:14.996 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.996 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:14.996 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5OO-o7lYlDap9QJLpGxlz4T", 2020-11-29T16:11:14.996 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:14.996 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.997 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:14.997 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:14.997 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.997 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:14.998 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:14.998 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:14.998 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.930041_148.1", 2020-11-29T16:11:14.998 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:14.998 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-43", 2020-11-29T16:11:14.999 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.930041631Z", 2020-11-29T16:11:14.999 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:14.999 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:14.999 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:14.999 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.000 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.000 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.000 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5OO-o7lYlDap9QJLpGxlz4T", 2020-11-29T16:11:15.000 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.000 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.001 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.002 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:15.002 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.002 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.002 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.002 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:15.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:15.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:15.005 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f053a404-d473-420f-9a01-38eedc856256 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:15.046 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:15.046 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:15.290 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.848433_141.1", 2020-11-29T16:11:15.291 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.292 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-42:3c165286-3738-4e0f-b399-acd859302998.5116.1", 2020-11-29T16:11:15.292 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.848433092Z", 2020-11-29T16:11:15.292 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.292 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.293 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.293 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.293 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.294 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_uJHlSItgASeLHf4vV6myM3g", 2020-11-29T16:11:15.294 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.294 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.294 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:15.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.296 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.852884_142.1", 2020-11-29T16:11:15.296 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.296 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-42:3c165286-3738-4e0f-b399-acd859302998.5116.1", 2020-11-29T16:11:15.296 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.852884445Z", 2020-11-29T16:11:15.297 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.297 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.297 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.297 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.298 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.298 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_uJHlSItgASeLHf4vV6myM3g", 2020-11-29T16:11:15.298 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.299 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.299 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:15.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.300 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.854739_143.1", 2020-11-29T16:11:15.300 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:15.301 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-42", 2020-11-29T16:11:15.301 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.854739278Z", 2020-11-29T16:11:15.301 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.301 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.302 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.302 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.302 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.303 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1OxpMJPB0AzK0o5_DAd_Re8", 2020-11-29T16:11:15.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.303 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.304 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:15.304 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.304 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.304 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.862141_144.1", 2020-11-29T16:11:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-42", 2020-11-29T16:11:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.862141657Z", 2020-11-29T16:11:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1OxpMJPB0AzK0o5_DAd_Re8", 2020-11-29T16:11:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.912191_145.1", 2020-11-29T16:11:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-43:3c165286-3738-4e0f-b399-acd859302998.5116.2", 2020-11-29T16:11:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.912191123Z", 2020-11-29T16:11:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tIFKXdQQP_3E_YrQvx0Z5eu", 2020-11-29T16:11:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.919404_146.1", 2020-11-29T16:11:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-43:3c165286-3738-4e0f-b399-acd859302998.5116.2", 2020-11-29T16:11:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.919404670Z", 2020-11-29T16:11:15.315 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.315 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.315 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.315 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tIFKXdQQP_3E_YrQvx0Z5eu", 2020-11-29T16:11:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.921398_147.1", 2020-11-29T16:11:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-43", 2020-11-29T16:11:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.921398751Z", 2020-11-29T16:11:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5OO-o7lYlDap9QJLpGxlz4T", 2020-11-29T16:11:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:15.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.930041_148.1", 2020-11-29T16:11:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-43", 2020-11-29T16:11:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.930041631Z", 2020-11-29T16:11:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5OO-o7lYlDap9QJLpGxlz4T", 2020-11-29T16:11:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.326 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.327 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:15.327 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.328 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:15.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:15.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:15.330 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f053a404-d473-420f-9a01-38eedc856256 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:15.368 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:15.369 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:15.602 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:15.603 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:15.603 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.848433_141.1", 2020-11-29T16:11:15.603 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.603 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-42:3c165286-3738-4e0f-b399-acd859302998.5116.1", 2020-11-29T16:11:15.603 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.848433092Z", 2020-11-29T16:11:15.604 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:15.604 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:15.604 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:15.604 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.605 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.605 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.605 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_uJHlSItgASeLHf4vV6myM3g", 2020-11-29T16:11:15.605 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.605 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.606 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.606 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:15.606 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.606 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.607 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.607 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:15.607 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.852884_142.1", 2020-11-29T16:11:15.607 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.607 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-42:3c165286-3738-4e0f-b399-acd859302998.5116.1", 2020-11-29T16:11:15.608 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.852884445Z", 2020-11-29T16:11:15.608 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:15.608 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:15.608 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:15.608 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.609 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.609 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.609 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_uJHlSItgASeLHf4vV6myM3g", 2020-11-29T16:11:15.609 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.610 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.610 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.610 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:15.610 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.610 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.611 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.611 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:15.611 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.854739_143.1", 2020-11-29T16:11:15.611 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:15.611 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-42", 2020-11-29T16:11:15.612 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.854739278Z", 2020-11-29T16:11:15.612 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:15.612 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:15.612 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:15.612 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.613 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.613 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.613 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1OxpMJPB0AzK0o5_DAd_Re8", 2020-11-29T16:11:15.613 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.613 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.614 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.614 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:15.614 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.614 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.615 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.615 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:15.615 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.862141_144.1", 2020-11-29T16:11:15.615 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:15.615 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-42", 2020-11-29T16:11:15.616 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.862141657Z", 2020-11-29T16:11:15.616 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:15.616 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:15.616 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:15.616 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.617 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.617 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.617 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_1OxpMJPB0AzK0o5_DAd_Re8", 2020-11-29T16:11:15.617 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.617 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.618 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.618 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:15.618 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.618 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.618 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.619 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:15.619 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.912191_145.1", 2020-11-29T16:11:15.619 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.619 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-43:3c165286-3738-4e0f-b399-acd859302998.5116.2", 2020-11-29T16:11:15.619 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.912191123Z", 2020-11-29T16:11:15.620 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:15.620 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:15.620 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:15.620 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.621 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.621 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.621 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tIFKXdQQP_3E_YrQvx0Z5eu", 2020-11-29T16:11:15.621 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.621 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.622 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.622 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:15.622 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.622 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.622 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.623 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:15.623 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.919404_146.1", 2020-11-29T16:11:15.623 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.623 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-43:3c165286-3738-4e0f-b399-acd859302998.5116.2", 2020-11-29T16:11:15.623 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.919404670Z", 2020-11-29T16:11:15.624 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:15.624 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:15.624 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:15.624 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.624 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.625 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.625 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_tIFKXdQQP_3E_YrQvx0Z5eu", 2020-11-29T16:11:15.625 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.625 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.626 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.626 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:15.626 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.626 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.626 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.627 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:15.627 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.921398_147.1", 2020-11-29T16:11:15.627 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:15.627 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-43", 2020-11-29T16:11:15.627 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.921398751Z", 2020-11-29T16:11:15.628 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:15.628 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:15.628 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:15.628 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.628 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.629 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.629 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5OO-o7lYlDap9QJLpGxlz4T", 2020-11-29T16:11:15.629 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.629 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.629 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.630 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:15.630 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.630 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.630 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.630 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:15.631 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666200.930041_148.1", 2020-11-29T16:11:15.631 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:15.631 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-43", 2020-11-29T16:11:15.631 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:10:00.930041631Z", 2020-11-29T16:11:15.631 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:15.632 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:15.632 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:15.632 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:15.632 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.633 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:15.633 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_5OO-o7lYlDap9QJLpGxlz4T", 2020-11-29T16:11:15.633 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:15.633 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:15.633 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:15.634 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:15.634 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.634 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.635 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:15.635 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:15.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:15.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:15.637 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f053a404-d473-420f-9a01-38eedc856256 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:15.677 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:15.677 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:15.911 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.848433_141.1", 2020-11-29T16:11:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-42:3c165286-3738-4e0f-b399-acd859302998.5116.1", 2020-11-29T16:11:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.848433092Z", 2020-11-29T16:11:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.914 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.914 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_uJHlSItgASeLHf4vV6myM3g", 2020-11-29T16:11:15.914 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.915 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.915 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:15.915 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.915 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.916 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.852884_142.1", 2020-11-29T16:11:15.916 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.916 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-42:3c165286-3738-4e0f-b399-acd859302998.5116.1", 2020-11-29T16:11:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.852884445Z", 2020-11-29T16:11:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.917 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_uJHlSItgASeLHf4vV6myM3g", 2020-11-29T16:11:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.919 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.919 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:15.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.920 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.854739_143.1", 2020-11-29T16:11:15.920 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:15.920 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-42", 2020-11-29T16:11:15.921 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.854739278Z", 2020-11-29T16:11:15.921 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.921 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.921 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.922 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.922 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1OxpMJPB0AzK0o5_DAd_Re8", 2020-11-29T16:11:15.922 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.923 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.923 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:15.923 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.923 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.924 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.862141_144.1", 2020-11-29T16:11:15.924 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:15.924 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-42", 2020-11-29T16:11:15.925 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.862141657Z", 2020-11-29T16:11:15.925 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.925 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.925 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.925 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.926 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.926 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1OxpMJPB0AzK0o5_DAd_Re8", 2020-11-29T16:11:15.926 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:15.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.928 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.912191_145.1", 2020-11-29T16:11:15.928 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.928 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-43:3c165286-3738-4e0f-b399-acd859302998.5116.2", 2020-11-29T16:11:15.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.912191123Z", 2020-11-29T16:11:15.929 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.929 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.929 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.930 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.930 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tIFKXdQQP_3E_YrQvx0Z5eu", 2020-11-29T16:11:15.930 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.931 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.931 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:15.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.932 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.919404_146.1", 2020-11-29T16:11:15.932 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:15.932 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-43:3c165286-3738-4e0f-b399-acd859302998.5116.2", 2020-11-29T16:11:15.932 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.919404670Z", 2020-11-29T16:11:15.933 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.933 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.933 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.933 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.934 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.934 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_tIFKXdQQP_3E_YrQvx0Z5eu", 2020-11-29T16:11:15.934 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.934 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.935 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:15.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.936 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.921398_147.1", 2020-11-29T16:11:15.936 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:15.936 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-43", 2020-11-29T16:11:15.936 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.921398751Z", 2020-11-29T16:11:15.936 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.937 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.937 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.937 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.937 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.938 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5OO-o7lYlDap9QJLpGxlz4T", 2020-11-29T16:11:15.938 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.938 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.938 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:15.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.939 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:15.939 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666200.930041_148.1", 2020-11-29T16:11:15.940 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:15.940 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-43", 2020-11-29T16:11:15.940 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:10:00.930041631Z", 2020-11-29T16:11:15.940 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:15.940 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:15.941 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:15.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:15.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.941 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:15.941 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5OO-o7lYlDap9QJLpGxlz4T", 2020-11-29T16:11:15.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:15.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:15.942 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:15.944 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:15.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:15.945 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:15.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:15.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:15.948 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 24836445-da90-4fa5-864b-eda958e6bd14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:15.990 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:15.990 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:16.217 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:16.217 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.217 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.184068_212.1", 2020-11-29T16:11:16.218 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.218 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-44:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1", 2020-11-29T16:11:16.218 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.184068523Z", 2020-11-29T16:11:16.218 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.219 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.219 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.219 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.219 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.219 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.220 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_RQKSdHtzmMxVqWgUX6QNn4S", 2020-11-29T16:11:16.220 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.220 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.220 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.221 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:16.221 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.221 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.221 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.221 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.222 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.188851_213.1", 2020-11-29T16:11:16.222 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.222 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-44:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1", 2020-11-29T16:11:16.222 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.188851572Z", 2020-11-29T16:11:16.222 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.223 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.223 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.223 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.223 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.223 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.224 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_RQKSdHtzmMxVqWgUX6QNn4S", 2020-11-29T16:11:16.224 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.224 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.224 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.224 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:16.225 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.225 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.225 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.225 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.225 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.191036_214.1", 2020-11-29T16:11:16.226 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:16.226 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-44", 2020-11-29T16:11:16.226 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.191036669Z", 2020-11-29T16:11:16.226 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.226 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.227 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.227 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.227 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.227 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.227 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "__DOuQV9-C5wNbCcvJz4Njop", 2020-11-29T16:11:16.228 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.228 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.228 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.228 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:16.228 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.229 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.229 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.229 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.229 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.195289_215.1", 2020-11-29T16:11:16.230 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:16.230 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-44", 2020-11-29T16:11:16.230 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.195289132Z", 2020-11-29T16:11:16.230 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.231 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.231 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.231 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.231 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.231 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.232 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "__DOuQV9-C5wNbCcvJz4Njop", 2020-11-29T16:11:16.232 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.232 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.232 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.232 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:16.233 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.233 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.233 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.233 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.233 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.235793_216.1", 2020-11-29T16:11:16.234 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.234 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-45:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2", 2020-11-29T16:11:16.234 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.235793780Z", 2020-11-29T16:11:16.234 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.234 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.235 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.235 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.235 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.235 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.236 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oKLWc_ApuReSas_dRIrexMn", 2020-11-29T16:11:16.236 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.236 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.236 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.236 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:16.237 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.237 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.237 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.237 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.237 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.242971_217.1", 2020-11-29T16:11:16.238 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.238 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-45:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2", 2020-11-29T16:11:16.238 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.242971481Z", 2020-11-29T16:11:16.238 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.238 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.239 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.239 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.239 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.239 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.239 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oKLWc_ApuReSas_dRIrexMn", 2020-11-29T16:11:16.240 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.240 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.240 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.240 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:16.240 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.241 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.241 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.241 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.241 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.245343_218.1", 2020-11-29T16:11:16.241 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:16.242 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-45", 2020-11-29T16:11:16.242 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.245343334Z", 2020-11-29T16:11:16.242 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.242 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.242 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.243 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.243 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.243 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.243 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_SGf-rMTr34J1dA7B1GVO48x", 2020-11-29T16:11:16.243 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.244 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.244 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.244 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:16.244 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.244 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.245 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.245 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.245 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.250986_219.1", 2020-11-29T16:11:16.245 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:16.246 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-45", 2020-11-29T16:11:16.246 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.250986937Z", 2020-11-29T16:11:16.246 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.246 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.247 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.247 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.247 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.247 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.247 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_SGf-rMTr34J1dA7B1GVO48x", 2020-11-29T16:11:16.248 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.248 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.248 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.250 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:16.250 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.251 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.251 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.251 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:16.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:16.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:16.254 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 24836445-da90-4fa5-864b-eda958e6bd14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:16.292 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:16.293 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:16.534 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:16.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:16.535 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.184068_212.1", 2020-11-29T16:11:16.535 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.535 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-44:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1", 2020-11-29T16:11:16.535 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.184068523Z", 2020-11-29T16:11:16.536 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:16.536 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:16.536 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:16.536 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:16.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.537 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:16.537 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_RQKSdHtzmMxVqWgUX6QNn4S", 2020-11-29T16:11:16.537 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:16.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:16.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:16.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:16.539 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.188851_213.1", 2020-11-29T16:11:16.539 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.540 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-44:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1", 2020-11-29T16:11:16.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.188851572Z", 2020-11-29T16:11:16.540 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:16.540 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:16.541 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:16.541 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:16.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.541 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:16.541 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_RQKSdHtzmMxVqWgUX6QNn4S", 2020-11-29T16:11:16.542 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:16.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.542 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:16.542 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:16.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:16.543 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.191036_214.1", 2020-11-29T16:11:16.544 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:16.544 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-44", 2020-11-29T16:11:16.544 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.191036669Z", 2020-11-29T16:11:16.544 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:16.545 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:16.545 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:16.545 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:16.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.546 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:16.546 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "__DOuQV9-C5wNbCcvJz4Njop", 2020-11-29T16:11:16.546 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:16.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.546 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:16.547 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:16.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.548 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:16.548 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.195289_215.1", 2020-11-29T16:11:16.549 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:16.549 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-44", 2020-11-29T16:11:16.549 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.195289132Z", 2020-11-29T16:11:16.549 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:16.549 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:16.550 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:16.550 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:16.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.550 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:16.551 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "__DOuQV9-C5wNbCcvJz4Njop", 2020-11-29T16:11:16.551 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:16.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.551 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:16.552 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:16.552 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.552 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.235793_216.1", 2020-11-29T16:11:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-45:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2", 2020-11-29T16:11:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.235793780Z", 2020-11-29T16:11:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oKLWc_ApuReSas_dRIrexMn", 2020-11-29T16:11:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.242971_217.1", 2020-11-29T16:11:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-45:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2", 2020-11-29T16:11:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.242971481Z", 2020-11-29T16:11:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oKLWc_ApuReSas_dRIrexMn", 2020-11-29T16:11:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:16.562 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.245343_218.1", 2020-11-29T16:11:16.562 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:16.562 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-45", 2020-11-29T16:11:16.562 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.245343334Z", 2020-11-29T16:11:16.562 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:16.563 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:16.563 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:16.563 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:16.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.563 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:16.564 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_SGf-rMTr34J1dA7B1GVO48x", 2020-11-29T16:11:16.564 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:16.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.564 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:16.565 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:16.565 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.565 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.565 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:16.566 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.250986_219.1", 2020-11-29T16:11:16.566 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:16.566 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-45", 2020-11-29T16:11:16.566 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.250986937Z", 2020-11-29T16:11:16.566 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:16.567 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:16.567 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:16.567 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:16.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.567 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:16.568 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_SGf-rMTr34J1dA7B1GVO48x", 2020-11-29T16:11:16.568 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:16.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:16.568 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:16.569 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:16.569 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.569 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.570 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:16.570 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:16.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:16.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:16.573 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 24836445-da90-4fa5-864b-eda958e6bd14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:16.614 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:16.614 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:16.863 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:16.864 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.864 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.184068_212.1", 2020-11-29T16:11:16.864 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.864 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-44:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1", 2020-11-29T16:11:16.865 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.184068523Z", 2020-11-29T16:11:16.865 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.865 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.865 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.865 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.866 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.866 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.866 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_RQKSdHtzmMxVqWgUX6QNn4S", 2020-11-29T16:11:16.866 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.867 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.867 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.867 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:16.867 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.868 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.868 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.868 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.868 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.188851_213.1", 2020-11-29T16:11:16.868 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.869 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-44:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1", 2020-11-29T16:11:16.869 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.188851572Z", 2020-11-29T16:11:16.869 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.869 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.870 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.870 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.870 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.870 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.870 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_RQKSdHtzmMxVqWgUX6QNn4S", 2020-11-29T16:11:16.871 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.871 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.871 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.871 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:16.871 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.872 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.872 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.872 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.872 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.191036_214.1", 2020-11-29T16:11:16.872 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:16.873 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-44", 2020-11-29T16:11:16.873 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.191036669Z", 2020-11-29T16:11:16.873 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.873 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.873 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.874 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.874 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.874 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.874 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "__DOuQV9-C5wNbCcvJz4Njop", 2020-11-29T16:11:16.875 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.875 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.875 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.875 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:16.875 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.876 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.876 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.876 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.876 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.195289_215.1", 2020-11-29T16:11:16.876 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:16.877 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-44", 2020-11-29T16:11:16.877 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.195289132Z", 2020-11-29T16:11:16.877 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.877 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.877 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.878 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.878 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.878 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.878 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "__DOuQV9-C5wNbCcvJz4Njop", 2020-11-29T16:11:16.879 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.879 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.879 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.879 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:16.880 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.880 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.880 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.880 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.880 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.235793_216.1", 2020-11-29T16:11:16.881 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.881 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-45:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2", 2020-11-29T16:11:16.881 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.235793780Z", 2020-11-29T16:11:16.881 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.882 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.882 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.882 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.882 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.882 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.883 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oKLWc_ApuReSas_dRIrexMn", 2020-11-29T16:11:16.883 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.883 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.883 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.883 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:16.884 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.884 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.884 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.884 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.884 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.242971_217.1", 2020-11-29T16:11:16.885 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket.instance", 2020-11-29T16:11:16.885 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-45:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2", 2020-11-29T16:11:16.885 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.242971481Z", 2020-11-29T16:11:16.885 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.885 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.886 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.886 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.886 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.886 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.887 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_oKLWc_ApuReSas_dRIrexMn", 2020-11-29T16:11:16.887 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.887 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.887 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.887 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:16.888 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.888 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.888 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.888 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.888 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.245343_218.1", 2020-11-29T16:11:16.889 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:16.889 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-45", 2020-11-29T16:11:16.889 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.245343334Z", 2020-11-29T16:11:16.889 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.889 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.890 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.890 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.890 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.890 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.891 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_SGf-rMTr34J1dA7B1GVO48x", 2020-11-29T16:11:16.891 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.891 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.891 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.891 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "write" 2020-11-29T16:11:16.892 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.892 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.892 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.892 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:16.892 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "1_1606666262.250986_219.1", 2020-11-29T16:11:16.893 INFO:teuthology.orchestra.run.smithi168.stdout: "section": "bucket", 2020-11-29T16:11:16.893 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-45", 2020-11-29T16:11:16.893 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:02.250986937Z", 2020-11-29T16:11:16.893 INFO:teuthology.orchestra.run.smithi168.stdout: "data": { 2020-11-29T16:11:16.893 INFO:teuthology.orchestra.run.smithi168.stdout: "read_version": { 2020-11-29T16:11:16.894 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "", 2020-11-29T16:11:16.894 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 0 2020-11-29T16:11:16.894 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.894 INFO:teuthology.orchestra.run.smithi168.stdout: "write_version": { 2020-11-29T16:11:16.894 INFO:teuthology.orchestra.run.smithi168.stdout: "tag": "_SGf-rMTr34J1dA7B1GVO48x", 2020-11-29T16:11:16.895 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": 1 2020-11-29T16:11:16.895 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:16.895 INFO:teuthology.orchestra.run.smithi168.stdout: "status": { 2020-11-29T16:11:16.896 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "complete" 2020-11-29T16:11:16.896 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.896 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.896 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:16.897 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:16.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:16.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:16.899 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 24836445-da90-4fa5-864b-eda958e6bd14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:16.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:16.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:17.170 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:17.170 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:17.171 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.184068_212.1", 2020-11-29T16:11:17.171 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:17.171 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-44:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1", 2020-11-29T16:11:17.171 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.184068523Z", 2020-11-29T16:11:17.172 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:17.172 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:17.172 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:17.172 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:17.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.173 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:17.173 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_RQKSdHtzmMxVqWgUX6QNn4S", 2020-11-29T16:11:17.173 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:17.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.174 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:17.174 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:17.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:17.175 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.188851_213.1", 2020-11-29T16:11:17.175 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:17.175 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-44:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1", 2020-11-29T16:11:17.176 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.188851572Z", 2020-11-29T16:11:17.176 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:17.176 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:17.176 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:17.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:17.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.177 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:17.177 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_RQKSdHtzmMxVqWgUX6QNn4S", 2020-11-29T16:11:17.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:17.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.178 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:17.178 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:17.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.180 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:17.180 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.191036_214.1", 2020-11-29T16:11:17.180 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:17.180 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-44", 2020-11-29T16:11:17.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.191036669Z", 2020-11-29T16:11:17.181 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:17.181 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:17.181 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:17.181 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:17.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.182 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:17.182 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "__DOuQV9-C5wNbCcvJz4Njop", 2020-11-29T16:11:17.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:17.184 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.195289_215.1", 2020-11-29T16:11:17.184 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:17.184 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-44", 2020-11-29T16:11:17.185 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.195289132Z", 2020-11-29T16:11:17.185 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:17.185 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:17.185 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:17.186 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:17.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.186 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:17.186 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "__DOuQV9-C5wNbCcvJz4Njop", 2020-11-29T16:11:17.186 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:17.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.187 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:17.187 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:17.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.235793_216.1", 2020-11-29T16:11:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-45:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2", 2020-11-29T16:11:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.235793780Z", 2020-11-29T16:11:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:17.190 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:17.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.190 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oKLWc_ApuReSas_dRIrexMn", 2020-11-29T16:11:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:17.193 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.242971_217.1", 2020-11-29T16:11:17.193 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-11-29T16:11:17.194 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-45:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2", 2020-11-29T16:11:17.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.242971481Z", 2020-11-29T16:11:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_oKLWc_ApuReSas_dRIrexMn", 2020-11-29T16:11:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.199 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:17.199 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.245343_218.1", 2020-11-29T16:11:17.199 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:17.199 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-45", 2020-11-29T16:11:17.200 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.245343334Z", 2020-11-29T16:11:17.200 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:17.200 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:17.200 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:17.200 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:17.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.201 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:17.201 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_SGf-rMTr34J1dA7B1GVO48x", 2020-11-29T16:11:17.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:17.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:17.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-11-29T16:11:17.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:17.203 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1606666262.250986_219.1", 2020-11-29T16:11:17.203 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-11-29T16:11:17.203 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-45", 2020-11-29T16:11:17.204 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:02.250986937Z", 2020-11-29T16:11:17.204 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-11-29T16:11:17.204 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-11-29T16:11:17.204 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-11-29T16:11:17.204 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-11-29T16:11:17.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.205 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-11-29T16:11:17.205 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_SGf-rMTr34J1dA7B1GVO48x", 2020-11-29T16:11:17.205 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-11-29T16:11:17.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:17.206 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-11-29T16:11:17.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-11-29T16:11:17.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:17.208 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:17.208 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:17.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:17.209 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:17.246 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:17.246 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:17.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:17.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:17.565 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:17.603 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:17.603 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:18.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:18.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:18.028 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:18.065 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:18.065 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:18.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:18.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:18.341 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:18.375 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:18.376 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:18.751 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:18.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:18.752 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e24cddf9-396c-4261-a7bc-b4e1e678325e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:18.792 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:18.792 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:19.073 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T16:11:19.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:19.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:19.077 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e24cddf9-396c-4261-a7bc-b4e1e678325e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:19.116 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:19.117 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:19.386 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-11-29T16:11:19.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:19.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:19.389 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e24cddf9-396c-4261-a7bc-b4e1e678325e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:19.430 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:19.430 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:19.696 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T16:11:19.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:19.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e24cddf9-396c-4261-a7bc-b4e1e678325e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:19.699 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e24cddf9-396c-4261-a7bc-b4e1e678325e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:19.737 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:19.738 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:19.994 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-11-29T16:11:19.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:19.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:19.997 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f053a404-d473-420f-9a01-38eedc856256 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:20.035 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:20.035 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:20.303 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T16:11:20.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:20.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:20.306 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f053a404-d473-420f-9a01-38eedc856256 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:20.345 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:20.345 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:20.606 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-11-29T16:11:20.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:20.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:20.610 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f053a404-d473-420f-9a01-38eedc856256 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:20.651 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:20.651 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:20.918 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T16:11:20.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:20.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f053a404-d473-420f-9a01-38eedc856256', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:20.921 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f053a404-d473-420f-9a01-38eedc856256 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:20.960 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:20.960 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:21.225 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-11-29T16:11:21.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:21.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:21.229 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 24836445-da90-4fa5-864b-eda958e6bd14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:21.269 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:21.269 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:21.520 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T16:11:21.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:21.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:21.523 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 24836445-da90-4fa5-864b-eda958e6bd14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:21.561 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:21.562 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:21.827 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-11-29T16:11:21.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:21.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:21.830 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 24836445-da90-4fa5-864b-eda958e6bd14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:21.870 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:21.870 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:22.140 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T16:11:22.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:22.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '24836445-da90-4fa5-864b-eda958e6bd14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:22.144 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 24836445-da90-4fa5-864b-eda958e6bd14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:22.182 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:22.182 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:22.444 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-11-29T16:11:22.447 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-11-29T16:11:22.448 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-46 2020-11-29T16:11:22.490 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-47 2020-11-29T16:11:22.574 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-48 2020-11-29T16:11:22.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:22.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:22.779 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:22.815 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:22.815 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:23.053 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:11:23.080 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:23.081 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:23.081 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:23.081 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:11:23.082 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:23.082 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:23.082 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:11:23.082 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:22.609250Z" 2020-11-29T16:11:23.082 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:23.083 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:23.083 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:23.083 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:11:23.083 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:23.083 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:23.084 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:23.084 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:11:23.084 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:23.084 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:23.085 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:11:23.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:23.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:23.087 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:23.131 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:23.131 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:23.379 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:23.379 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:11:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:11:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:11:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:11:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:11:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:11:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:11:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:11:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:11:23.383 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:23.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:11:23.383 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:23.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:23.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:23.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:23.384 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:11:23.384 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:23.384 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:23.384 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:11:23.385 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:23.385 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:23.385 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:23.385 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:11:23.385 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:23.386 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:23.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:23.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:23.386 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:11:23.386 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:11:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:23.387 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:11:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:23.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:11:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:11:23.389 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:11:23.390 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:23.391 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:11:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:11:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:11:23.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:23.392 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:23.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:11:23.394 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:11:23.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:23.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:23.395 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:23.433 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:23.433 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:23.678 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:11:23.678 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:11:23.678 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:11:23.679 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:11:23.679 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:11:23.679 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:11:23.679 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:11:23.680 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:23.680 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:11:23.680 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:23.680 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:11:23.680 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:23.681 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:11:23.681 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:11:23.681 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:23.681 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:11:23.681 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:23.682 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:11:23.682 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:11:23.682 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:23.682 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:23.682 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:23.683 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:11:23.683 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:23.683 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:11:23.683 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:11:23.683 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:23.684 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:23.684 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:23.684 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:11:23.684 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:11:23.684 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:23.685 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:23.685 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:23.685 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:11:23.685 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:23.685 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:11:23.686 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:23.686 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:23.686 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:23.686 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:23.686 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:11:23.687 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:11:23.687 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:23.687 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:23.687 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:23.687 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:11:23.687 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:23.688 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:11:23.688 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:11:23.688 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:23.688 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:23.688 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:23.689 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:11:23.689 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:11:23.689 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:23.689 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:23.689 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:11:23.690 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:23.690 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:11:23.690 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:11:23.690 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:11:23.690 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:23.691 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:11:23.691 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:11:23.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:11:23.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:23.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:23.693 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:23.735 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:23.735 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:24.001 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:24.001 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:11:24.001 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:11:24.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:11:24.002 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:11:24.002 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:11:24.002 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:24.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.003 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:11:24.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:24.003 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:11:24.003 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:24.003 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:24.004 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:11:24.004 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:24.004 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:11:24.004 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:24.004 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:11:24.005 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:24.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:24.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:11:24.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:24.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:24.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:11:24.006 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:24.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:24.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:24.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:11:24.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:24.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:24.008 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:11:24.008 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:24.008 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:24.009 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:11:24.009 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:24.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:24.009 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:24.009 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:11:24.010 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:24.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:24.010 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:11:24.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:24.011 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:11:24.011 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:11:24.011 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:24.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:24.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:24.012 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:11:24.012 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:11:24.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.013 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:24.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.013 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:11:24.013 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:11:24.014 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:11:24.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.014 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:24.015 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:11:24.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:24.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:24.016 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:24.057 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:24.057 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:24.310 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:24.310 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:24.311 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:11:24.311 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:11:24.311 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:24.311 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:24.311 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:11:24.312 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:11:24.312 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:24.312 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:24.312 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:24.312 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:11:24.313 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:24.313 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:24.313 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:24.313 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:11:24.313 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:24.314 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:24.315 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:11:24.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:24.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:24.316 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:24.359 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:24.359 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:24.645 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:24.645 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:11:24.646 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:11:24.646 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:11:24.646 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:11:24.646 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 6162550666273865027 2020-11-29T16:11:24.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.647 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:11:24.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:24.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:11:24.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:24.648 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:24.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:11:24.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:24.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:24.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:24.649 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:11:24.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:24.649 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:11:24.650 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:24.650 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:24.650 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:11:24.650 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:24.651 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:24.651 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:24.651 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:11:24.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:11:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:24.652 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:11:24.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:24.654 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:11:24.655 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:24.655 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:24.655 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:24.655 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:24.656 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:24.656 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:24.656 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:11:24.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.657 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:24.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:24.657 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:11:24.657 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:11:24.657 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:11:24.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:24.658 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:24.659 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:11:24.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:24.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:24.660 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:24.700 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:24.700 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:24.979 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:24.979 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:24.980 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:11:24.980 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:11:24.980 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:24.980 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:24.981 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:11:24.981 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:11:24.981 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:24.981 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:24.982 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:24.982 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:11:24.982 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:24.982 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:24.983 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:24.983 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:11:24.983 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:24.983 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:24.984 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:11:24.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:24.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:24.986 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:25.067 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:25.067 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:25.344 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:11:25.344 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:11:25.344 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:11:25.345 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:11:25.345 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:11:25.345 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 5268709521422156746 2020-11-29T16:11:25.346 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:25.346 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:11:25.346 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:25.347 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:11:25.347 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:25.347 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:25.347 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:11:25.348 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:25.348 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:25.348 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:25.348 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:11:25.349 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:25.349 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:25.349 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:25.349 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:11:25.350 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:25.350 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:25.350 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:11:25.350 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:25.351 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:25.351 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:25.351 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:11:25.351 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:25.351 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:25.352 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:25.352 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:11:25.352 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:25.352 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:25.353 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:25.353 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:25.353 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:25.353 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:25.354 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:11:25.354 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:25.354 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:25.354 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:25.355 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:11:25.355 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:25.355 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:25.355 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:25.356 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:25.356 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:25.356 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:25.356 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:11:25.357 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:25.357 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:25.357 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:11:25.357 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:25.358 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:11:25.358 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:11:25.358 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:11:25.358 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:25.359 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:11:25.360 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:11:25.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:25.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:25.361 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:25.444 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:25.444 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:25.696 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:25.696 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:25.696 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:11:25.697 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:11:25.697 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:25.697 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:25.697 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:11:25.697 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:11:25.698 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:25.698 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:25.698 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:25.698 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:11:25.698 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:25.699 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:25.699 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:25.699 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:11:25.699 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:25.700 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:25.700 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:11:25.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:25.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:25.702 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:25.744 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:25.745 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:26.012 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:11:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:11:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:11:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:11:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 16089441476439148043 2020-11-29T16:11:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:11:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:11:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:11:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:11:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:11:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:11:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:11:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:11:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:11:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:11:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:11:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:11:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:11:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:11:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:26.025 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:26.026 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:11:26.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:26.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:26.027 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:26.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:26.109 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:26.376 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:11:22.984434Z" 2020-11-29T16:11:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:11:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:08:47.474921Z" 2020-11-29T16:11:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:11:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:11:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:26.380 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:26.381 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:11:26.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:26.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:26.383 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:26.426 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:26.426 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:26.684 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:11:26.685 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:11:26.685 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:11:26.685 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:11:26.685 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:11:26.686 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 201246450793848048 2020-11-29T16:11:26.686 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:26.686 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:11:26.686 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:26.686 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:11:26.687 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:26.687 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:26.687 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:26.687 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:26.688 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:26.688 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:26.688 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:22.984434Z" 2020-11-29T16:11:26.688 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:26.688 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:26.689 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:26.689 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:11:26.689 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:26.689 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:26.689 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:26.690 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:26.690 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:26.690 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:26.690 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:11:26.690 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:26.691 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:26.691 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:26.691 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:11:26.691 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:26.691 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:26.692 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:26.692 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:26.692 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:26.692 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:26.692 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.474921Z" 2020-11-29T16:11:26.693 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:26.693 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:26.693 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:26.693 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:11:26.693 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:26.694 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:26.694 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:11:26.694 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:26.694 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:26.694 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:26.695 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:11:26.695 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:26.695 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:26.695 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:11:26.695 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:26.696 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:11:26.696 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:11:26.696 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:11:26.696 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:26.696 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:11:26.697 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:11:26.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:26.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:26.699 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:26.740 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:26.999 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:26.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:27.000 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:27.000 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:11:22.984434Z" 2020-11-29T16:11:27.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:27.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:27.000 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:27.001 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:11:27.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:27.001 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:27.001 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:27.001 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:08:47.474921Z" 2020-11-29T16:11:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:11:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:11:27.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:27.003 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:27.004 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:11:27.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:27.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:27.005 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:27.047 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:27.047 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:27.297 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:11:27.298 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:11:27.298 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:11:27.298 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:11:27.298 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:11:27.299 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 16876182154419717581 2020-11-29T16:11:27.299 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.299 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:11:27.299 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.299 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:11:27.300 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:27.300 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:27.300 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:27.300 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:27.301 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:27.301 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:27.301 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:22.984434Z" 2020-11-29T16:11:27.301 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.301 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.302 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.302 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:11:27.302 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:27.302 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:27.302 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:27.303 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:27.303 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:27.303 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:27.303 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:11:27.303 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.304 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.304 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.304 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:11:27.304 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:27.305 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:27.305 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:27.305 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:27.305 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:27.305 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:27.306 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.474921Z" 2020-11-29T16:11:27.306 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.306 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.306 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.306 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:11:27.307 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:27.307 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:27.307 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:11:27.307 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:27.307 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:27.308 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:27.308 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:11:27.308 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.308 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.309 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:11:27.309 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.309 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:11:27.309 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:11:27.309 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:11:27.310 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.310 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:11:27.311 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:11:27.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:27.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:27.312 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:27.399 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:27.399 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:27.660 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:27.660 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.661 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:27.661 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:23.024075Z" 2020-11-29T16:11:27.661 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.661 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.662 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:27.662 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.461995Z" 2020-11-29T16:11:27.662 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.662 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.662 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:27.663 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:47.412069Z" 2020-11-29T16:11:27.663 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.663 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.663 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:27.664 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:57.000149Z" 2020-11-29T16:11:27.664 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.664 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:27.665 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:11:27.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:27.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:27.667 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:27.707 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:27.707 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:27.974 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:11:27.975 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:11:27.975 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:11:27.975 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:11:27.975 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:11:27.975 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 3114293326551352523 2020-11-29T16:11:27.976 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.976 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:11:27.976 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.976 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:11:27.976 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:27.977 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:27.977 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:27.977 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:27.977 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:27.977 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:27.978 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:11:23.024075Z" 2020-11-29T16:11:27.978 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.978 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.978 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.979 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:11:27.979 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:27.979 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:27.979 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:27.979 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:27.980 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:27.980 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:27.980 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:04:59.461995Z" 2020-11-29T16:11:27.980 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.980 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.981 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.981 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:11:27.981 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:27.981 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:27.981 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:27.982 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:27.982 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:27.982 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:27.982 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:08:47.412069Z" 2020-11-29T16:11:27.982 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.983 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.983 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:27.983 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:11:27.983 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:11:27.983 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:11:27.984 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:27.984 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:11:27.984 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:11:27.984 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:11:27.985 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:03:57.000149Z" 2020-11-29T16:11:27.985 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.985 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.985 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:11:27.985 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:27.985 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:11:27.986 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:11:27.986 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:11:27.986 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:27.986 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:11:27.987 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:11:27.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:27.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:27.989 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:28.076 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:28.076 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:28.345 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:28.345 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:28.346 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:28.346 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:23.024075Z" 2020-11-29T16:11:28.346 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:28.346 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:28.346 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:28.347 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.461995Z" 2020-11-29T16:11:28.347 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:28.347 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:28.347 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:28.348 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:47.412069Z" 2020-11-29T16:11:28.348 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:28.348 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:28.348 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:28.348 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:57.000149Z" 2020-11-29T16:11:28.349 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:28.349 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:28.350 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:11:28.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:28.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:28.351 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:28.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:28.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:28.664 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:28.664 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:11:28.664 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:11:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:11:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:11:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 2748347558345283993 2020-11-29T16:11:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:11:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:11:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:23.024075Z" 2020-11-29T16:11:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:11:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:04:59.461995Z" 2020-11-29T16:11:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:11:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.412069Z" 2020-11-29T16:11:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:11:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:11:28.674 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:28.674 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:11:28.674 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:11:28.674 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:11:28.675 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:03:57.000149Z" 2020-11-29T16:11:28.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:28.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:28.675 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:28.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:28.676 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:11:28.676 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:11:28.676 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:11:28.676 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:28.676 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:28.677 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:11:28.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:28.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:28.679 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:28.716 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:28.717 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:28.978 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:28.978 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:28.978 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:11:28.979 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:11:28.979 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:28.979 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:28.979 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:11:28.980 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:11:28.980 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:28.980 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:28.980 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:28.980 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:11:28.981 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:28.981 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:28.981 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:28.981 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:11:28.982 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:28.982 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:28.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:28.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:28.984 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:29.061 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:29.061 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:29.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:29.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:29.330 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:29.368 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:29.369 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:29.633 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:11:29.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:29.634 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-11-29T16:11:29.634 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:11:22.984434Z" 2020-11-29T16:11:29.634 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:29.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:29.635 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005034", 2020-11-29T16:11:29.635 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:04:37.450640Z" 2020-11-29T16:11:29.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:29.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:29.636 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:29.636 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:08:47.474921Z" 2020-11-29T16:11:29.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:29.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:29.637 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002640", 2020-11-29T16:11:29.637 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:03:53.446876Z" 2020-11-29T16:11:29.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:29.637 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:11:29.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:29.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:29.640 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:29.716 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:29.716 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:29.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:29.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:11:29.999 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:11:30.036 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:30.036 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:30.328 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:11:30.328 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:30.328 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:11:30.328 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:11:30.329 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:30.329 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:30.329 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005699", 2020-11-29T16:11:30.329 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:04:59.315700Z" 2020-11-29T16:11:30.330 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:30.330 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:30.330 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:11:30.330 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:11:30.331 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:11:30.331 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:11:30.331 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:11:30.331 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:11:30.331 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:11:30.332 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:11:30.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:30.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:30.334 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:30.409 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:30.409 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:30.698 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-11-29T16:11:30.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:30.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:30.700 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:30.742 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:30.743 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:30.774 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:11:30.770+0000 7f6279113080 0 failed reading obj info from .rgw.root:zone_info.36679b47-164f-436b-b728-5846b32cea58: (2) No such file or directory 2020-11-29T16:11:30.774 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:11:30.770+0000 7f6279113080 0 WARNING: could not read zone params for zone id=36679b47-164f-436b-b728-5846b32cea58 name=test-zone3 2020-11-29T16:11:30.779 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:11:30.774+0000 7f6279113080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T16:11:30.779 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:11:30.774+0000 7f6279113080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T16:11:30.794 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:11:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:11:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-29T16:11:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-29T16:11:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-29T16:11:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-29T16:11:30.795 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-29T16:11:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-29T16:11:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-29T16:11:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-29T16:11:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-29T16:11:30.796 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-29T16:11:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-29T16:11:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-29T16:11:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-29T16:11:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-29T16:11:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T16:11:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T16:11:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T16:11:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T16:11:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T16:11:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:11:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-29T16:11:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T16:11:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T16:11:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-29T16:11:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T16:11:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-29T16:11:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T16:11:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:11:30.802 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-29T16:11:30.802 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:30.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:30.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:11:30.804 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:11:30.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:11:30.884 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:11:31.185 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T16:11:31.257 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:11:31.258 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:11:31.258 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2, 2020-11-29T16:11:31.258 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:11:31.258 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-11-29T16:11:31.259 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:11:31.259 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:11:31.259 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:11:31.259 INFO:teuthology.orchestra.run.smithi184.stdout: "" 2020-11-29T16:11:31.260 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.260 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-11-29T16:11:31.260 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:11:31.260 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-11-29T16:11:31.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.261 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-11-29T16:11:31.261 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-11-29T16:11:31.261 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-11-29T16:11:31.261 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-11-29T16:11:31.261 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:11:31.262 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:11:31.262 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.262 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-11-29T16:11:31.262 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-11-29T16:11:31.263 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:11:31.263 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-11-29T16:11:31.263 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.263 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:11:31.264 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-11-29T16:11:31.264 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:11:31.265 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:11:31.265 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.265 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:11:31.265 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:11:31.265 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:11:31.266 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:11:31.266 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:11:31.266 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:11:31.266 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:11:31.266 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:11:31.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.267 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.267 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:11:31.267 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:11:31.267 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:11:31.268 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:11:31.268 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.268 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:11:31.268 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:11:31.268 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:11:31.269 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:11:31.269 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:11:31.269 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T16:11:31.269 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:11:31.270 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:11:31.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.270 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.270 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:11:31.270 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T16:11:31.271 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:11:31.271 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:11:31.271 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.271 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:11:31.271 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:11:31.272 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:11:31.272 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:11:31.272 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-11-29T16:11:31.272 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T16:11:31.272 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-11-29T16:11:31.273 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-11-29T16:11:31.273 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.273 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:11:31.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.274 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:11:31.274 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-11-29T16:11:31.274 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:11:31.275 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:11:31.275 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.275 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:11:31.275 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:11:31.275 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:11:31.276 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:11:31.276 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:11:31.276 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:11:31.276 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:11:31.276 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:11:31.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:31.277 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.277 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-11-29T16:11:31.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.278 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-11-29T16:11:31.278 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-11-29T16:11:31.278 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-11-29T16:11:31.278 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-11-29T16:11:31.279 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:31.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:31.279 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.279 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-11-29T16:11:31.279 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:11:31.280 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-11-29T16:11:31.280 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-11-29T16:11:31.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:31.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:31.280 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:11:31.281 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-11-29T16:11:31.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:11:31.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 791374490 2020-11-29T16:11:31.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:11:31.283 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2491292681 2020-11-29T16:11:31.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.283 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:11:31.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 293306859 2020-11-29T16:11:31.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.284 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:11:31.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:11:31.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1785095687 2020-11-29T16:11:31.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:31.285 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:11:31.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.285 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:11:31.285 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:11:31.286 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-11-29T16:11:31.286 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-11-29T16:11:31.286 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:11:31.286 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:11:31.286 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:11:31.287 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:11:31.287 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:11:31.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.287 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-11-29T16:11:31.287 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:11:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:11:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:11:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:11:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:11:31.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:11:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:11:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:11:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-11-29T16:11:31.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:11:31.289 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:11:31.290 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-11-29T16:12:31.292 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-49 2020-11-29T16:12:31.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:31.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:31.396 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:12:31.433 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:12:31.434 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:12:31.673 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:12:31.707 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:12:31.707 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:12:31.707 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:12:31.708 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:12:31.708 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:12:31.708 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:12:31.708 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:12:31.708 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:12:31.336452Z" 2020-11-29T16:12:31.709 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:12:31.709 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:12:31.709 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:12:31.709 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:12:31.710 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:12:31.710 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:12:31.710 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:12:31.710 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:12:31.710 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:12:31.711 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:12:31.712 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:12:31.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:31.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:31.713 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:12:31.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:12:31.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:12:32.014 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:12:32.014 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:12:32.015 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:12:32.015 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:12:32.015 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:12:32.015 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:12:32.016 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:32.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:32.016 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:12:32.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:32.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:12:32.017 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:32.017 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:32.017 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:12:32.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:32.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:12:32.018 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:32.018 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:12:32.018 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:32.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:32.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:32.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:32.019 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:12:32.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:32.020 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:32.020 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:12:32.020 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:32.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:32.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:32.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:12:32.021 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:32.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:32.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:32.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:32.022 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:12:32.022 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:32.023 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:32.023 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:32.023 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:32.023 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:32.024 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:32.024 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:32.024 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:32.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:32.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:32.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:32.026 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:12:32.026 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:32.026 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:32.027 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:32.027 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:32.027 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:32.027 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:32.028 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:32.028 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:32.028 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:32.028 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:32.029 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:12:32.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:32.029 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:12:32.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:12:32.029 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:12:32.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:32.030 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:12:32.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:12:37.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:37.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:37.034 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:12:37.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:12:37.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:12:37.336 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:12:37.337 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:12:37.337 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:12:37.337 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:12:37.337 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:12:37.338 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:12:37.338 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:37.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:37.338 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:12:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:12:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:12:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:12:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:12:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:12:37.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:37.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:37.342 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:12:37.342 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:12:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:12:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:12:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:12:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:12:37.350 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:12:37.350 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:12:37.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:37.350 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:12:37.351 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:12:42.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:42.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:42.354 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:12:42.395 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:12:42.395 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:12:42.661 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:12:42.662 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:12:42.662 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:12:42.662 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:12:42.662 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:12:42.663 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:12:42.663 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:42.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:42.663 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:12:42.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:42.664 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:12:42.664 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:42.664 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:42.664 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:12:42.665 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:42.665 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:12:42.665 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:42.665 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:12:42.666 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:42.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:42.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:42.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:42.666 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:12:42.667 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:42.667 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:42.667 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:12:42.667 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:42.668 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:42.668 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:42.668 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:12:42.668 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:42.668 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:42.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:42.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:42.669 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:12:42.669 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:42.670 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:42.670 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:42.670 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:42.670 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:42.670 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:42.671 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:42.671 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:42.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:42.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:42.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:42.672 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:12:42.672 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:42.672 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:42.672 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:42.672 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:42.673 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:42.673 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:42.673 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:42.673 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:42.674 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:42.674 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:42.674 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:12:42.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:42.674 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:12:42.675 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:12:42.675 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:12:42.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:42.675 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:12:42.676 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:12:47.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:47.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:47.678 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:12:47.719 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:12:47.719 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:12:47.987 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:12:47.987 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:12:47.987 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:12:47.988 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:12:47.988 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:12:47.988 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:12:47.988 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:47.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:47.989 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:12:47.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:47.989 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:12:47.989 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:47.990 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:47.990 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:12:47.990 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:47.990 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:12:47.990 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:47.991 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:12:47.991 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:47.991 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:47.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:47.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:47.992 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:12:47.992 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:47.992 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:47.992 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:12:47.993 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:47.993 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:47.993 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:47.993 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:12:47.993 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:47.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:47.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:47.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:47.995 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:12:47.995 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:47.995 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:47.995 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:47.995 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:47.996 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:12:47.997 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:47.998 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:47.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:12:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:12:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:12:48.000 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:12:48.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:48.001 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:12:48.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:12:53.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:53.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:53.004 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:12:53.046 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:12:53.046 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:12:53.331 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:12:53.331 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:12:53.331 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:12:53.332 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:12:53.332 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:12:53.332 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:12:53.332 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:53.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:53.333 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:12:53.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:53.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:12:53.334 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:53.334 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:53.334 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:12:53.334 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:53.334 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:12:53.335 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:53.335 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:12:53.335 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:53.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:12:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:53.336 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:53.337 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:12:53.337 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:53.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:53.337 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:53.337 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:12:53.338 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:53.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:53.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:53.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:53.339 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:12:53.339 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:53.339 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:53.339 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:53.340 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:53.340 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:53.340 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:53.340 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:53.340 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:12:53.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:12:58.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:58.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:12:58.348 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:12:58.390 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:12:58.390 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:12:58.670 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:12:58.670 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:12:58.670 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:12:58.671 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:12:58.671 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:12:58.671 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:12:58.672 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:58.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:58.672 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:12:58.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:58.672 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:12:58.673 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:58.673 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:58.673 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:12:58.673 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:58.673 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:12:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:12:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:58.674 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:58.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:58.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:58.675 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:12:58.675 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:58.675 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:58.676 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:12:58.676 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:58.676 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:58.676 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:58.676 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:12:58.677 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:12:58.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:58.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:58.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:58.678 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:12:58.678 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:58.678 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:58.678 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:58.678 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:58.679 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:58.679 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:58.679 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:58.679 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:58.679 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:58.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:58.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:12:58.680 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:12:58.680 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:12:58.680 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:12:58.681 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:12:58.681 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:12:58.681 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:12:58.681 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:12:58.681 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:12:58.682 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:12:58.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:58.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:58.682 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:12:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:12:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:12:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:12:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:12:58.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:12:58.684 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:12:58.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:13:02.828 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:13:02.825+0000 7fec017ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:13:03.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:03.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:03.687 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:03.729 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:03.729 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:04.008 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:04.009 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:13:04.009 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:13:04.009 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:13:04.010 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:13:04.010 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:04.011 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:04.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:04.012 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:13:04.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:04.012 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:13:04.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:04.013 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:04.014 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:04.014 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:04.014 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:13:04.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:04.015 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:04.016 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:04.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:04.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:04.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:04.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:13:04.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:04.018 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:04.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:13:04.019 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:04.019 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:04.020 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:04.020 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:13:04.020 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:04.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:04.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:04.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:04.022 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:13:04.022 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:04.023 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:04.023 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:04.024 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:04.024 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:04.024 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:04.025 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:04.025 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:04.026 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:04.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:04.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:04.027 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:13:04.027 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:04.028 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:04.028 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:04.028 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:04.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:04.029 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:04.030 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:04.030 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:04.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:04.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:04.031 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:04.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:04.032 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:13:04.032 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:13:04.033 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:13:04.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:04.033 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:04.034 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:13:09.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:09.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:09.037 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:09.080 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:09.080 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:09.342 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:09.343 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:13:09.343 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:13:09.343 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:13:09.343 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:13:09.344 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:09.344 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:09.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:09.344 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:13:09.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:09.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:13:09.345 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:09.345 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:09.345 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:09.346 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:09.346 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:13:09.346 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:09.346 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:09.347 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:09.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:09.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:09.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:09.348 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:13:09.348 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:09.348 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:09.348 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:13:09.349 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:09.349 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:09.349 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:09.349 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:13:09.350 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:09.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:09.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:09.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:09.350 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:13:09.351 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:09.351 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:09.351 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:09.351 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:09.352 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:09.352 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:09.352 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:09.352 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:09.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:09.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:09.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:09.353 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:13:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:09.354 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:09.355 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:09.355 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:09.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:09.355 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:09.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:09.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:09.356 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:09.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:09.356 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:13:09.357 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:13:09.357 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:13:09.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:09.357 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:09.359 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:13:14.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:14.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:14.362 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:14.402 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:14.403 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:14.679 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:14.680 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:13:14.680 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:13:14.680 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:13:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:13:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:14.681 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:13:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:13:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:14.682 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:13:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:13:14.685 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:14.685 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:14.685 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:13:14.685 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:13:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:14.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:14.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:14.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:14.687 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:13:14.687 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:14.688 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:14.688 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:14.688 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:14.688 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:14.688 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:14.689 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:14.689 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:14.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:14.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:14.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:14.690 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:13:14.690 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:14.690 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:14.690 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:14.691 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:14.691 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:14.691 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:14.691 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:14.691 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:14.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:14.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:14.692 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:14.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:14.693 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:13:14.693 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:13:14.693 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:13:14.693 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:14.693 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:14.694 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:13:19.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:19.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:19.696 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:19.740 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:19.741 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:20.012 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:20.013 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:13:20.013 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:13:20.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:13:20.014 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:13:20.014 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:20.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:20.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:20.015 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:13:20.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:20.015 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:13:20.015 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:20.015 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:20.016 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:20.016 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:20.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:13:20.016 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:20.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:20.017 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:20.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:20.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:20.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:20.018 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:13:20.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:20.018 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:20.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:13:20.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:20.019 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:20.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:20.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:13:20.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:20.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:20.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:20.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:20.020 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:13:20.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:20.021 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:20.021 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:20.021 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:20.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:20.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:20.022 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:20.022 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:20.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:20.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:20.023 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:20.023 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:13:20.023 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:20.023 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:20.023 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:20.024 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:20.024 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:20.024 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:20.024 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:20.025 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:20.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:20.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:20.025 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:20.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:20.026 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:13:20.026 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:13:20.026 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:13:20.026 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:20.027 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:20.028 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:13:25.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:25.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:25.031 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:25.071 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:25.071 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:25.332 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:25.333 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:13:25.333 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:13:25.333 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:13:25.334 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:13:25.334 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:25.334 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:25.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:25.334 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:13:25.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:25.335 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:13:25.335 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:25.335 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:25.336 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:25.336 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:25.336 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:13:25.336 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:25.336 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:25.337 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:25.337 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:25.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:25.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:25.337 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:13:25.338 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:25.338 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:25.338 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:13:25.338 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:25.339 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:25.339 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:25.339 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:13:25.339 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:25.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:25.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:25.340 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:25.340 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:13:25.340 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:25.341 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:25.341 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:25.341 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:25.341 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:25.341 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:25.342 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:25.342 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:25.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:25.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:25.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:25.343 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:13:25.343 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:25.343 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:25.344 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:25.344 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:25.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:25.344 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:25.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:25.345 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:25.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:25.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:25.345 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:25.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:25.346 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:13:25.346 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:13:25.346 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:13:25.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:25.347 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:25.347 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:13:30.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:30.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:30.350 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:30.391 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:30.391 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:30.682 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:30.682 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:13:30.683 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:13:30.683 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:13:30.683 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:13:30.683 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:30.683 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:30.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:30.684 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:13:30.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:30.684 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:13:30.685 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:30.685 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:30.685 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:30.685 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:30.685 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:13:30.686 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:30.686 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:30.686 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:30.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:30.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:30.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:30.687 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:13:30.687 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:30.687 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:30.688 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666282.609250_234.1", 2020-11-29T16:13:30.688 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:30.688 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:30.688 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:30.688 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:22.609250Z", 2020-11-29T16:13:30.689 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:30.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:30.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:30.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:30.690 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:13:30.690 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:30.690 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:30.690 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:30.690 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:30.691 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:13:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:30.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:30.695 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:13:30.695 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:13:30.695 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:13:30.695 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:30.696 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:30.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666351.336452_248.1 target=1_1606666282.609250_234.1 2020-11-29T16:13:31.484 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T16:13:31.479+0000 7f9a8a7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:13:35.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:35.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:35.699 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:35.741 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:35.741 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:36.008 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:36.009 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:13:36.009 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:13:36.009 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:13:36.010 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:13:36.010 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:36.010 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:36.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.010 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:13:36.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.011 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:13:36.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.011 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:36.012 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:36.012 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:36.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:13:36.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:36.012 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:36.013 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:36.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.014 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:13:36.014 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.014 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:36.014 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:13:36.014 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:36.015 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:36.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:36.015 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:13:36.015 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:36.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.016 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:13:36.016 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.016 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:36.017 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:36.017 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:36.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:36.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:36.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:36.018 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:36.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.019 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:13:36.019 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.019 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:36.019 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:36.019 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:36.020 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:36.020 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:36.020 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:36.020 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:36.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.021 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:36.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.021 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:13:36.022 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:13:36.022 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:13:36.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.022 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:36.023 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:13:36.024 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:13:36.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:36.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:36.025 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:36.065 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:36.065 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:36.311 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:13:36.312 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:13:36.312 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:13:36.312 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:13:36.312 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:13:36.312 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:36.313 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:13:36.313 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:13:36.313 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:13:36.313 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:13:36.314 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:13:36.314 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:13:36.314 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:13:36.314 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:36.315 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:13:36.315 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:13:36.315 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:13:36.315 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:36.315 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:13:36.316 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:13:36.316 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:13:36.316 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:13:36.316 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:13:36.317 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:13:36.317 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:13:36.317 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:13:36.317 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:13:36.317 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:13:36.318 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:13:36.318 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:13:36.318 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:13:36.318 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:13:36.318 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:13:36.319 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:13:36.319 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:13:36.319 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:13:36.319 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:13:36.319 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:13:36.320 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:13:36.320 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:13:36.320 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:13:36.320 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:13:36.321 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:13:36.321 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:13:36.321 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:13:36.321 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:13:36.321 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:13:36.322 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:13:36.322 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:13:36.322 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:13:36.322 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:13:36.322 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:13:36.323 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:13:36.323 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:13:36.323 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:13:36.323 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:13:36.324 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:13:36.324 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:13:36.324 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:13:36.324 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:13:36.324 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:13:36.325 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:13:36.325 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:13:36.325 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:13:36.326 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:13:36.327 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:13:36.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:36.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:13:36.328 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:13:36.369 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:36.369 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:36.736 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:36.737 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:13:36.737 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:13:36.737 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:13:36.738 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:13:36.738 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:36.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:36.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.739 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:13:36.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.739 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:13:36.739 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.739 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:36.740 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:13:36.740 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:36.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:13:36.740 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:36.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:13:36.741 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:36.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:13:36.742 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.742 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:36.742 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:13:36.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:36.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:36.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:36.743 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:13:36.743 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:36.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.744 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:13:36.745 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.745 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:36.745 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:36.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:36.746 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:36.746 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:36.746 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:36.746 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:36.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.747 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:13:36.747 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.748 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:13:36.748 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:13:36.748 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:13:36.748 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:13:36.748 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:13:36.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:13:36.749 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:13:36.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:13:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:13:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:13:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.751 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:36.752 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:13:36.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=a1734b2a-fbe5-446c-a86c-68fcf5b64a76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:13:36.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=a1734b2a-fbe5-446c-a86c-68fcf5b64a76', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:13:36.772 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:13:36.831 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:36.832 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:36.862 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:13:36.859+0000 7fcbd9cea080 0 failed reading obj info from .rgw.root:zone_info.36679b47-164f-436b-b728-5846b32cea58: (2) No such file or directory 2020-11-29T16:13:36.862 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:13:36.859+0000 7fcbd9cea080 0 WARNING: could not read zone params for zone id=36679b47-164f-436b-b728-5846b32cea58 name=test-zone3 2020-11-29T16:13:36.865 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:13:36.863+0000 7fcbd9cea080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T16:13:36.865 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:13:36.863+0000 7fcbd9cea080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T16:13:36.876 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:36.877 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:13:36.877 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:13:36.877 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-29T16:13:36.877 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-29T16:13:36.878 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-29T16:13:36.878 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-29T16:13:36.878 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-29T16:13:36.878 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-29T16:13:36.879 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-29T16:13:36.879 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-29T16:13:36.879 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-29T16:13:36.879 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-29T16:13:36.880 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-29T16:13:36.880 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-29T16:13:36.880 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-29T16:13:36.880 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-29T16:13:36.880 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T16:13:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T16:13:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T16:13:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.881 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T16:13:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T16:13:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:13:36.882 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-29T16:13:36.883 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T16:13:36.883 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T16:13:36.883 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-29T16:13:36.883 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T16:13:36.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:36.884 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-29T16:13:36.884 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T16:13:36.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.885 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:36.885 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:36.885 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:13:36.885 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-29T16:13:36.886 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:36.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:13:36.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:13:36.888 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:13:36.963 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:13:36.963 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:13:37.248 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T16:13:37.321 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:13:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:37.323 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 3, 2020-11-29T16:13:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:13:37.324 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-11-29T16:13:37.325 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:13:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:13:37.326 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:13:37.327 INFO:teuthology.orchestra.run.smithi184.stdout: "" 2020-11-29T16:13:37.328 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.329 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-11-29T16:13:37.329 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:13:37.330 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-11-29T16:13:37.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.332 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-11-29T16:13:37.332 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-11-29T16:13:37.333 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-11-29T16:13:37.334 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-11-29T16:13:37.335 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:13:37.335 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:13:37.336 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.336 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-11-29T16:13:37.337 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-11-29T16:13:37.337 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:13:37.337 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-11-29T16:13:37.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:13:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-11-29T16:13:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:13:37.339 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:13:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:13:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:13:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:13:37.340 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:13:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:13:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:13:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:13:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:13:37.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.342 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:13:37.342 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:13:37.342 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:13:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:13:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:13:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:13:37.343 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:13:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:13:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:13:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T16:13:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:13:37.344 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76" 2020-11-29T16:13:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:13:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T16:13:37.345 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:13:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:13:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:13:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:13:37.346 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:13:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:13:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-11-29T16:13:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T16:13:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-11-29T16:13:37.347 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-11-29T16:13:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:13:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.348 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:13:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-11-29T16:13:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:13:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:13:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.349 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:13:37.350 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:13:37.350 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:13:37.350 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:13:37.350 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:13:37.350 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:13:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:13:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:13:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.351 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-11-29T16:13:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-11-29T16:13:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-11-29T16:13:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-11-29T16:13:37.352 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-11-29T16:13:37.353 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:37.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:37.353 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.353 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-11-29T16:13:37.353 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:13:37.354 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-11-29T16:13:37.354 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-11-29T16:13:37.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:37.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:37.355 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:13:37.356 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-11-29T16:13:37.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.356 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:13:37.356 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 791374490 2020-11-29T16:13:37.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.357 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:13:37.357 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2491292681 2020-11-29T16:13:37.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.358 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:13:37.358 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 293306859 2020-11-29T16:13:37.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.358 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:13:37.358 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:13:37.359 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1785095687 2020-11-29T16:13:37.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:37.359 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:13:37.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.359 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:13:37.360 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:13:37.360 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-11-29T16:13:37.360 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-11-29T16:13:37.360 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:13:37.361 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:13:37.361 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:13:37.361 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:13:37.361 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:13:37.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.362 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-11-29T16:13:37.362 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:13:37.362 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:13:37.362 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:13:37.362 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:13:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:13:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:13:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:13:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:13:37.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-11-29T16:13:37.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:13:37.364 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:13:37.364 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "a1734b2a-fbe5-446c-a86c-68fcf5b64a76" 2020-11-29T16:14:37.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:14:37.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:14:37.716 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:14:37.755 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:14:37.755 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:14:37.785 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:14:37.781+0000 7f9509381080 0 failed reading obj info from .rgw.root:zone_info.36679b47-164f-436b-b728-5846b32cea58: (2) No such file or directory 2020-11-29T16:14:37.785 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:14:37.781+0000 7f9509381080 0 WARNING: could not read zone params for zone id=36679b47-164f-436b-b728-5846b32cea58 name=test-zone3 2020-11-29T16:14:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:14:37.781+0000 7f9509381080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T16:14:37.787 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:14:37.781+0000 7f9509381080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T16:14:37.797 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:14:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:14:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:14:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-29T16:14:37.797 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-29T16:14:37.798 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-29T16:14:37.798 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-29T16:14:37.798 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-29T16:14:37.798 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-29T16:14:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-29T16:14:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-29T16:14:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-29T16:14:37.799 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-29T16:14:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-29T16:14:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-29T16:14:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-29T16:14:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-29T16:14:37.800 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T16:14:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T16:14:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T16:14:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T16:14:37.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T16:14:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:14:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-29T16:14:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T16:14:37.802 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T16:14:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-29T16:14:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T16:14:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:37.803 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-29T16:14:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T16:14:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:37.804 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:14:37.805 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-29T16:14:37.805 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:14:37.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:14:37.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:14:37.807 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:14:38.160 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T16:14:38.235 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:14:38.236 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:14:38.236 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 4, 2020-11-29T16:14:38.236 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:14:38.236 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-11-29T16:14:38.237 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:14:38.237 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:14:38.237 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:14:38.237 INFO:teuthology.orchestra.run.smithi184.stdout: "" 2020-11-29T16:14:38.237 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.238 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-11-29T16:14:38.238 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:14:38.238 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-11-29T16:14:38.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.239 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-11-29T16:14:38.239 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-11-29T16:14:38.239 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-11-29T16:14:38.239 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-11-29T16:14:38.239 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:14:38.240 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:14:38.240 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.240 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-11-29T16:14:38.240 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-11-29T16:14:38.240 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:14:38.241 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-11-29T16:14:38.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.241 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:14:38.241 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-11-29T16:14:38.241 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:14:38.242 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:14:38.242 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.242 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:14:38.242 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:14:38.242 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:14:38.243 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:14:38.243 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:14:38.243 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:14:38.243 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:14:38.243 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:14:38.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.244 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:14:38.244 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:14:38.245 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:14:38.245 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:14:38.245 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.245 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:14:38.245 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:14:38.246 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:14:38.246 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:14:38.246 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:14:38.246 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:14:38.247 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:14:38.247 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76" 2020-11-29T16:14:38.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.247 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:14:38.248 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T16:14:38.248 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:14:38.248 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:14:38.248 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.248 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:14:38.249 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:14:38.249 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:14:38.249 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:14:38.249 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-11-29T16:14:38.249 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T16:14:38.250 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-11-29T16:14:38.250 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-11-29T16:14:38.250 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.250 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:14:38.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.251 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.251 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:14:38.251 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-11-29T16:14:38.251 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:14:38.252 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:14:38.252 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.252 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:14:38.252 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:14:38.252 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:14:38.253 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:14:38.253 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:14:38.253 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:14:38.253 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:14:38.253 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:14:38.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:38.254 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.254 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-11-29T16:14:38.254 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.254 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-11-29T16:14:38.255 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-11-29T16:14:38.255 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-11-29T16:14:38.255 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-11-29T16:14:38.255 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:14:38.255 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:38.256 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.256 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-11-29T16:14:38.256 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:14:38.256 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-11-29T16:14:38.256 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-11-29T16:14:38.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:38.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:38.257 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:14:38.258 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-11-29T16:14:38.258 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.258 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:14:38.258 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 791374490 2020-11-29T16:14:38.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.259 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.259 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:14:38.259 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2491292681 2020-11-29T16:14:38.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.260 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:14:38.260 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 293306859 2020-11-29T16:14:38.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:14:38.261 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:14:38.261 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1785095687 2020-11-29T16:14:38.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:38.261 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:14:38.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.262 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:14:38.262 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:14:38.262 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-11-29T16:14:38.262 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-11-29T16:14:38.263 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:14:38.263 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:14:38.263 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:14:38.263 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:14:38.263 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:14:38.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.264 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-11-29T16:14:38.264 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:14:38.264 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:14:38.264 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:14:38.265 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:14:38.265 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:14:38.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:14:38.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:14:38.265 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:14:38.266 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-11-29T16:14:38.266 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:14:38.266 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:14:38.267 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-11-29T16:15:38.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:15:38.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:15:38.270 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:15:38.310 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:15:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:15:38.342 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:15:38.336+0000 7fa773234080 0 failed reading obj info from .rgw.root:zone_info.36679b47-164f-436b-b728-5846b32cea58: (2) No such file or directory 2020-11-29T16:15:38.342 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:15:38.336+0000 7fa773234080 0 WARNING: could not read zone params for zone id=36679b47-164f-436b-b728-5846b32cea58 name=test-zone3 2020-11-29T16:15:38.345 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:15:38.340+0000 7fa773234080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T16:15:38.345 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:15:38.340+0000 7fa773234080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T16:15:38.356 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:15:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:15:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:15:38.356 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-11-29T16:15:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-11-29T16:15:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-11-29T16:15:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-11-29T16:15:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-11-29T16:15:38.357 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-11-29T16:15:38.358 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-11-29T16:15:38.358 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-11-29T16:15:38.358 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-11-29T16:15:38.358 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-11-29T16:15:38.358 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-11-29T16:15:38.359 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-11-29T16:15:38.359 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-11-29T16:15:38.359 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-11-29T16:15:38.359 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T16:15:38.359 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T16:15:38.360 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T16:15:38.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.360 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T16:15:38.360 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.360 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T16:15:38.361 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:15:38.361 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-11-29T16:15:38.361 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T16:15:38.361 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T16:15:38.361 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-11-29T16:15:38.362 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-11-29T16:15:38.362 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.362 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-11-29T16:15:38.363 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T16:15:38.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.363 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:15:38.364 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-11-29T16:15:38.364 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:15:38.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:15:38.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:15:38.366 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:15:38.443 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:15:38.443 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:15:38.718 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T16:15:38.793 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:15:38.794 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:15:38.794 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 5, 2020-11-29T16:15:38.794 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:15:38.795 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-11-29T16:15:38.795 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:15:38.795 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:15:38.795 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:15:38.796 INFO:teuthology.orchestra.run.smithi184.stdout: "" 2020-11-29T16:15:38.796 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.796 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-11-29T16:15:38.796 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:15:38.797 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-11-29T16:15:38.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.797 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-11-29T16:15:38.797 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-11-29T16:15:38.797 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-11-29T16:15:38.798 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-11-29T16:15:38.798 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:15:38.798 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:15:38.798 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.799 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-11-29T16:15:38.799 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-11-29T16:15:38.799 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:15:38.799 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-11-29T16:15:38.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.800 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:15:38.800 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-11-29T16:15:38.800 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:15:38.801 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:15:38.801 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.801 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:15:38.801 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:15:38.802 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:15:38.802 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:15:38.802 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:15:38.802 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:15:38.802 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:15:38.803 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:15:38.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.803 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:15:38.804 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:15:38.804 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:15:38.804 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:15:38.804 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.805 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:15:38.805 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:15:38.805 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:15:38.805 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:15:38.806 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:15:38.806 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:15:38.806 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:15:38.806 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:15:38.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.807 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.807 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:15:38.807 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T16:15:38.807 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:15:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:15:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:15:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:15:38.808 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:15:38.809 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:15:38.809 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-11-29T16:15:38.809 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T16:15:38.809 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-11-29T16:15:38.809 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-11-29T16:15:38.810 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.810 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:15:38.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.811 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:15:38.811 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-11-29T16:15:38.811 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:15:38.811 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:15:38.812 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.812 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:15:38.812 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:15:38.812 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:15:38.813 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:15:38.813 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:15:38.813 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:15:38.813 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:15:38.813 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:15:38.814 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.814 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.814 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-11-29T16:15:38.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.815 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-11-29T16:15:38.815 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-11-29T16:15:38.815 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-11-29T16:15:38.815 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-11-29T16:15:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:15:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.816 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-11-29T16:15:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:15:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-11-29T16:15:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-11-29T16:15:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.818 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:15:38.820 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-11-29T16:15:38.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.821 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:15:38.822 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 791374490 2020-11-29T16:15:38.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.823 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:15:38.823 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2491292681 2020-11-29T16:15:38.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.824 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:15:38.824 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 293306859 2020-11-29T16:15:38.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:15:38.825 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:15:38.825 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1785095687 2020-11-29T16:15:38.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.825 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:15:38.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.826 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:15:38.826 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:15:38.827 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-11-29T16:15:38.827 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-11-29T16:15:38.827 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:15:38.827 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:15:38.827 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:15:38.828 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:15:38.828 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:15:38.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.828 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-11-29T16:15:38.829 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:15:38.829 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:15:38.829 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:15:38.829 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:15:38.829 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:15:38.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:15:38.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:15:38.830 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:15:38.830 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-11-29T16:15:38.831 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:15:38.831 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:15:38.832 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-11-29T16:16:38.833 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-11-29T16:16:38.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:38.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:38.836 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:16:38.876 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:38.877 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:39.098 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:16:39.133 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:16:39.133 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:39.133 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:16:39.133 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:16:39.134 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:39.134 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:39.134 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:16:39.134 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:12:31.336452Z" 2020-11-29T16:16:39.135 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:39.135 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:39.135 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:16:39.135 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:16:39.135 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:39.136 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:39.136 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:16:39.136 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:16:39.136 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:39.137 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:16:39.137 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:16:39.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:39.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:39.140 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:16:39.181 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:39.181 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:39.442 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:16:39.442 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:16:39.442 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:16:39.443 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:16:39.443 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:16:39.443 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:39.443 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:16:39.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:39.444 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:16:39.444 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:39.444 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:16:39.444 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:39.444 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:39.445 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:16:39.445 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:39.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:16:39.445 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:39.445 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:16:39.446 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:39.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:39.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:39.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:39.447 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:16:39.447 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:39.447 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:39.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:16:39.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:39.448 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:39.448 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:39.448 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:16:39.448 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:16:39.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:16:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:16:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:16:39.451 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:39.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:39.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:39.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:39.451 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:16:39.452 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:39.452 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:39.452 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:16:39.452 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:39.452 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:39.453 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:39.453 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:16:39.453 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:39.453 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:39.453 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:39.454 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:16:39.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:39.454 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:16:39.454 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:16:39.454 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:16:39.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:39.455 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:16:39.456 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:16:39.456 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:16:39.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:39.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:39.458 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:16:39.500 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:39.500 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:39.757 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:16:39.758 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:16:39.758 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:16:39.758 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:16:39.759 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:16:39.759 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:39.759 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:16:39.759 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:39.760 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:16:39.760 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:39.760 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:16:39.760 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:16:39.760 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:16:39.761 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:16:39.761 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:16:39.761 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:16:39.761 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:16:39.762 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:16:39.762 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:16:39.762 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:39.762 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:39.762 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:39.763 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:16:39.763 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:16:39.763 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:16:39.763 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:16:39.763 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:16:39.764 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:16:39.764 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:16:39.764 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:16:39.764 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:16:39.764 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:39.765 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:39.765 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:39.765 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:16:39.765 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:16:39.766 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:16:39.766 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:16:39.766 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:16:39.766 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:16:39.766 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:16:39.767 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:16:39.767 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:16:39.767 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:39.767 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:39.767 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:39.768 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:16:39.768 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:16:39.768 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:16:39.768 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:16:39.768 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:16:39.769 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:16:39.769 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:16:39.769 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:16:39.769 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:16:39.769 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:39.770 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:39.770 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:39.770 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:39.770 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:16:39.771 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:16:39.771 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:16:39.771 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:39.771 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:16:39.772 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:16:39.773 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:16:39.773 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:39.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:39.774 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:16:39.812 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:39.813 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:40.182 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:16:40.183 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:16:40.183 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:16:40.184 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:16:40.184 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:16:40.185 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:40.185 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:16:40.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:40.186 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:16:40.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:40.187 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:16:40.187 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:40.188 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:40.188 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:16:40.188 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:40.189 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:16:40.189 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:40.190 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:16:40.190 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:40.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:40.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:40.191 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:16:40.192 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:40.192 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:40.193 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:16:40.193 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:16:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:16:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:16:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:16:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:16:40.200 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:40.200 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:40.201 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:40.201 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:16:40.202 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:40.202 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:40.203 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:16:40.203 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:40.204 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:40.204 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:40.204 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:16:40.205 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:40.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.206 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:16:40.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:40.207 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:16:40.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:16:40.208 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:16:40.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.209 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:16:40.211 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:16:40.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:40.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'R0Z42LAJFCIVZ75M', '--secret', 'Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:40.213 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key R0Z42LAJFCIVZ75M --secret Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:40.297 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:40.298 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:40.355 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:40.350+0000 7f4c0e001080 0 failed reading obj info from .rgw.root:zone_info.36679b47-164f-436b-b728-5846b32cea58: (2) No such file or directory 2020-11-29T16:16:40.356 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:40.350+0000 7f4c0e001080 0 WARNING: could not read zone params for zone id=36679b47-164f-436b-b728-5846b32cea58 name=test-zone3 2020-11-29T16:16:40.357 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:40.350+0000 7f4c0e001080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T16:16:40.358 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:40.350+0000 7f4c0e001080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T16:16:40.366 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:16:40.366 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "dacbb93d-ccb4-42b3-91da-23d210a79a64", 2020-11-29T16:16:40.367 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "remove", 2020-11-29T16:16:40.367 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "remove.rgw.meta:root", 2020-11-29T16:16:40.368 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "remove.rgw.control", 2020-11-29T16:16:40.368 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "remove.rgw.log:gc", 2020-11-29T16:16:40.369 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "remove.rgw.log:lc", 2020-11-29T16:16:40.369 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "remove.rgw.log", 2020-11-29T16:16:40.369 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-11-29T16:16:40.370 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-11-29T16:16:40.370 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-11-29T16:16:40.371 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-11-29T16:16:40.371 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-11-29T16:16:40.371 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-11-29T16:16:40.372 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-11-29T16:16:40.372 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-11-29T16:16:40.373 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "remove.rgw.otp", 2020-11-29T16:16:40.373 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-11-29T16:16:40.373 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "R0Z42LAJFCIVZ75M", 2020-11-29T16:16:40.374 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "Xug4XGVzHMaZ6V22EWOnLgdiHPJv74wx" 2020-11-29T16:16:40.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:40.375 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-11-29T16:16:40.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:40.376 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-11-29T16:16:40.376 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:40.376 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "remove.rgw.buckets.index", 2020-11-29T16:16:40.377 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-11-29T16:16:40.377 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-11-29T16:16:40.378 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "remove.rgw.buckets.data" 2020-11-29T16:16:40.378 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:40.379 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-11-29T16:16:40.379 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-11-29T16:16:40.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:40.381 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:40.381 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:40.381 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "remove.rgw.log:notif" 2020-11-29T16:16:40.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:16:40.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:40.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:40.386 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:40.464 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:40.465 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:48.021 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone a1734b2a-fbe5-446c-a86c-68fcf5b64a76 2020-11-29T16:16:48.092 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:16:48.092 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:48.093 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 6, 2020-11-29T16:16:48.093 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:16:48.094 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-11-29T16:16:48.094 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:16:48.094 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:16:48.095 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-11-29T16:16:48.095 INFO:teuthology.orchestra.run.smithi184.stdout: "" 2020-11-29T16:16:48.096 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.096 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-11-29T16:16:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-11-29T16:16:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.097 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-11-29T16:16:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-11-29T16:16:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-11-29T16:16:48.098 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-11-29T16:16:48.099 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:16:48.099 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.099 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.099 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-11-29T16:16:48.100 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-11-29T16:16:48.100 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.100 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-11-29T16:16:48.101 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.101 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:48.101 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-11-29T16:16:48.101 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:16:48.101 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:16:48.102 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.102 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:16:48.102 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:16:48.102 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.103 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:16:48.103 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:16:48.103 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.104 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:16:48.104 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:16:48.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.104 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.104 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:48.105 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-11-29T16:16:48.105 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:16:48.105 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.105 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.106 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:16:48.106 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:16:48.106 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.107 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:16:48.107 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:16:48.107 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.107 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:16:48.107 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:16:48.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.108 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.108 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:48.108 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-11-29T16:16:48.108 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:16:48.109 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:16:48.109 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.109 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:16:48.109 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:16:48.109 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.110 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:16:48.110 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-11-29T16:16:48.110 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-11-29T16:16:48.110 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-11-29T16:16:48.111 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-11-29T16:16:48.111 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.111 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:16:48.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.111 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.112 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.112 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-11-29T16:16:48.112 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-11-29T16:16:48.112 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.112 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.113 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:16:48.113 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:16:48.113 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.113 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:16:48.113 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:16:48.114 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.114 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:16:48.114 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:16:48.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.115 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.115 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "dacbb93d-ccb4-42b3-91da-23d210a79a64", 2020-11-29T16:16:48.115 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "remove", 2020-11-29T16:16:48.115 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [], 2020-11-29T16:16:48.115 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-11-29T16:16:48.116 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-11-29T16:16:48.116 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.116 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-11-29T16:16:48.116 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-11-29T16:16:48.116 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.117 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-11-29T16:16:48.117 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-11-29T16:16:48.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:48.118 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.118 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-11-29T16:16:48.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.119 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-11-29T16:16:48.119 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-11-29T16:16:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-11-29T16:16:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-11-29T16:16:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:16:48.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:48.121 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.121 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-11-29T16:16:48.121 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:48.121 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-11-29T16:16:48.122 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-11-29T16:16:48.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:48.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:48.122 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-11-29T16:16:48.122 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-11-29T16:16:48.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.123 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:48.123 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 791374490 2020-11-29T16:16:48.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.124 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:48.124 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2491292681 2020-11-29T16:16:48.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.125 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:48.125 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 293306859 2020-11-29T16:16:48.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.126 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.126 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.126 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1785095687 2020-11-29T16:16:48.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:48.127 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "dacbb93d-ccb4-42b3-91da-23d210a79a64", 2020-11-29T16:16:48.127 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 3660802685 2020-11-29T16:16:48.127 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:48.128 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:16:48.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.128 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:16:48.128 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.128 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-11-29T16:16:48.129 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-11-29T16:16:48.129 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:16:48.129 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:16:48.129 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:16:48.130 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:16:48.130 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:16:48.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.130 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-11-29T16:16:48.130 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-11-29T16:16:48.131 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-11-29T16:16:48.131 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-11-29T16:16:48.131 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-11-29T16:16:48.131 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-11-29T16:16:48.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:48.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:48.132 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:48.132 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-11-29T16:16:48.132 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:16:48.133 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:16:48.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.134 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:48.176 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:48.177 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:48.201 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:16:48.193+0000 7f74cb5c7080 0 failed reading obj info from .rgw.root:zone_info.3c165286-3738-4e0f-b399-acd859302998: (2) No such file or directory 2020-11-29T16:16:48.202 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:16:48.193+0000 7f74cb5c7080 0 WARNING: could not read zone params for zone id=3c165286-3738-4e0f-b399-acd859302998 name=test-zone2 2020-11-29T16:16:48.202 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:16:48.193+0000 7f74cb5c7080 0 failed reading obj info from .rgw.root:zone_info.82b01792-db79-401e-8298-c419db80f2a2: (2) No such file or directory 2020-11-29T16:16:48.203 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:16:48.193+0000 7f74cb5c7080 0 WARNING: could not read zone params for zone id=82b01792-db79-401e-8298-c419db80f2a2 name=test-zone4 2020-11-29T16:16:48.208 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:16:48.209 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "test-zonegroup", 2020-11-29T16:16:48.209 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zonegroup", 2020-11-29T16:16:48.210 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "", 2020-11-29T16:16:48.210 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "true", 2020-11-29T16:16:48.210 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.211 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.211 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.212 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T16:16:48.212 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T16:16:48.212 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.213 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [ 2020-11-29T16:16:48.213 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.213 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:48.213 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone3", 2020-11-29T16:16:48.214 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.214 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:16:48.214 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.214 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:48.215 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:48.215 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.215 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:48.215 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:48.216 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.216 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:48.216 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:48.217 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.217 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.217 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:48.217 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone2", 2020-11-29T16:16:48.217 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.218 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.218 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.218 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:48.218 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:48.219 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.219 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:48.219 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:48.219 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.219 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:48.220 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:48.220 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.220 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.220 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:48.221 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone4", 2020-11-29T16:16:48.221 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.221 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:16:48.221 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.221 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:48.222 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:48.222 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.222 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:48.222 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "pubsub", 2020-11-29T16:16:48.223 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "false", 2020-11-29T16:16:48.223 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [ 2020-11-29T16:16:48.223 INFO:teuthology.orchestra.run.smithi168.stdout: "test-zone1" 2020-11-29T16:16:48.224 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.224 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:48.225 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.225 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.226 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.226 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T16:16:48.226 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.227 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.227 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.228 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:48.228 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:48.229 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.229 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:48.229 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:48.230 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.230 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:48.231 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:48.231 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.232 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.232 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [ 2020-11-29T16:16:48.233 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.233 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default-placement", 2020-11-29T16:16:48.233 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-11-29T16:16:48.234 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": [ 2020-11-29T16:16:48.234 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD" 2020-11-29T16:16:48.234 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:48.235 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.235 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.236 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "default-placement", 2020-11-29T16:16:48.236 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:48.236 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T16:16:48.237 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T16:16:48.237 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.238 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:16:48.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.241 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:48.326 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:48.326 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:48.360 INFO:teuthology.orchestra.run.smithi168.stderr:zone name remove not found in zonegroup test-zonegroup 2020-11-29T16:16:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-11-29T16:16:48.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.366 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:48.409 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:48.449 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:48.442+0000 7f2775dc1080 0 failed reading obj info from .rgw.root:zone_info.36679b47-164f-436b-b728-5846b32cea58: (2) No such file or directory 2020-11-29T16:16:48.449 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:48.442+0000 7f2775dc1080 0 WARNING: could not read zone params for zone id=36679b47-164f-436b-b728-5846b32cea58 name=test-zone3 2020-11-29T16:16:48.451 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:48.446+0000 7f2775dc1080 0 failed reading obj info from .rgw.root:zone_info.a1734b2a-fbe5-446c-a86c-68fcf5b64a76: (2) No such file or directory 2020-11-29T16:16:48.451 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:48.446+0000 7f2775dc1080 0 WARNING: could not read zone params for zone id=a1734b2a-fbe5-446c-a86c-68fcf5b64a76 name=test-zone1 2020-11-29T16:16:48.456 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:16:48.450+0000 7f2775dc1080 0 zone id dacbb93d-ccb4-42b3-91da-23d210a79a64 is not a part of zonegroup default 2020-11-29T16:16:48.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.469 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:48.508 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:48.508 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:48.569 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:16:48.561+0000 7f414d6aa080 0 WARNING: can't generate connection for zone dacbb93d-ccb4-42b3-91da-23d210a79a64 id remove: no endpoints defined 2020-11-29T16:16:48.858 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:16:48.859 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:48.859 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 7, 2020-11-29T16:16:48.860 INFO:teuthology.orchestra.run.smithi168.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:16:48.860 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": [ 2020-11-29T16:16:48.861 INFO:teuthology.orchestra.run.smithi168.stdout: "", 2020-11-29T16:16:48.861 INFO:teuthology.orchestra.run.smithi168.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:16:48.861 INFO:teuthology.orchestra.run.smithi168.stdout: "", 2020-11-29T16:16:48.862 INFO:teuthology.orchestra.run.smithi168.stdout: "" 2020-11-29T16:16:48.862 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.863 INFO:teuthology.orchestra.run.smithi168.stdout: "period_map": { 2020-11-29T16:16:48.863 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:48.864 INFO:teuthology.orchestra.run.smithi168.stdout: "zonegroups": [ 2020-11-29T16:16:48.864 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.864 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "test-zonegroup", 2020-11-29T16:16:48.865 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zonegroup", 2020-11-29T16:16:48.865 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "", 2020-11-29T16:16:48.865 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "true", 2020-11-29T16:16:48.866 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.866 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.866 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.867 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T16:16:48.867 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T16:16:48.867 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.868 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [ 2020-11-29T16:16:48.868 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.868 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:48.868 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone3", 2020-11-29T16:16:48.869 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.869 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:16:48.869 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.869 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:48.870 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:48.870 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.870 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:48.870 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:48.871 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.871 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:48.871 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:48.871 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.872 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.872 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:48.872 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone2", 2020-11-29T16:16:48.872 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.872 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.873 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.873 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:48.873 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:48.873 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.873 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:48.874 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:48.874 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.874 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:48.874 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:48.875 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.875 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.875 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:48.875 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone4", 2020-11-29T16:16:48.876 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.876 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:16:48.876 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.876 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:48.876 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:48.877 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.877 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:48.877 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "pubsub", 2020-11-29T16:16:48.877 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "false", 2020-11-29T16:16:48.877 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [ 2020-11-29T16:16:48.878 INFO:teuthology.orchestra.run.smithi168.stdout: "test-zone1" 2020-11-29T16:16:48.878 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.878 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:48.878 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.878 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.879 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.879 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T16:16:48.879 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:48.879 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:48.880 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.880 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:48.880 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:48.880 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:48.880 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:48.881 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:48.881 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:48.881 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:48.881 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:48.882 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.882 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.882 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [ 2020-11-29T16:16:48.882 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.882 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default-placement", 2020-11-29T16:16:48.883 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-11-29T16:16:48.883 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": [ 2020-11-29T16:16:48.883 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD" 2020-11-29T16:16:48.883 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:48.883 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.884 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.884 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "default-placement", 2020-11-29T16:16:48.884 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:48.884 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T16:16:48.885 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T16:16:48.885 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.885 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.885 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:48.898 INFO:teuthology.orchestra.run.smithi168.stdout: "short_zone_ids": [ 2020-11-29T16:16:48.898 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.899 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:48.899 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 791374490 2020-11-29T16:16:48.899 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.899 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.900 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:48.900 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 2491292681 2020-11-29T16:16:48.900 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.900 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.901 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:48.901 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 293306859 2020-11-29T16:16:48.901 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.901 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:48.901 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.902 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 1785095687 2020-11-29T16:16:48.902 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.902 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:48.902 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.902 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:16:48.903 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:48.903 INFO:teuthology.orchestra.run.smithi168.stdout: "period_config": { 2020-11-29T16:16:48.903 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_quota": { 2020-11-29T16:16:48.903 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T16:16:48.903 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T16:16:48.904 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T16:16:48.904 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T16:16:48.904 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T16:16:48.904 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.904 INFO:teuthology.orchestra.run.smithi168.stdout: "user_quota": { 2020-11-29T16:16:48.905 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T16:16:48.905 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T16:16:48.905 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T16:16:48.905 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T16:16:48.905 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T16:16:48.906 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:48.906 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:48.906 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:48.906 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_name": "test-realm", 2020-11-29T16:16:48.907 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:16:48.907 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:16:48.908 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-11-29T16:16:48.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:48.910 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:48.989 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:48.989 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:49.022 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:16:49.022 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "5494521b-4bb4-4524-99ab-1276129abed5", 2020-11-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "remove_zg", 2020-11-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "remove_zg", 2020-11-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "false", 2020-11-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [], 2020-11-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "", 2020-11-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [], 2020-11-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [], 2020-11-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "", 2020-11-29T16:16:49.025 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:49.025 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T16:16:49.025 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T16:16:49.025 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.025 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:16:49.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.027 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:49.105 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:49.105 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:49.177 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:16:49.178 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "9e93b063-604d-4129-bc49-ebead4c467c6", 2020-11-29T16:16:49.178 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "remove", 2020-11-29T16:16:49.178 INFO:teuthology.orchestra.run.smithi168.stdout: "domain_root": "remove.rgw.meta:root", 2020-11-29T16:16:49.178 INFO:teuthology.orchestra.run.smithi168.stdout: "control_pool": "remove.rgw.control", 2020-11-29T16:16:49.179 INFO:teuthology.orchestra.run.smithi168.stdout: "gc_pool": "remove.rgw.log:gc", 2020-11-29T16:16:49.179 INFO:teuthology.orchestra.run.smithi168.stdout: "lc_pool": "remove.rgw.log:lc", 2020-11-29T16:16:49.179 INFO:teuthology.orchestra.run.smithi168.stdout: "log_pool": "remove.rgw.log", 2020-11-29T16:16:49.179 INFO:teuthology.orchestra.run.smithi168.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-11-29T16:16:49.180 INFO:teuthology.orchestra.run.smithi168.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-11-29T16:16:49.180 INFO:teuthology.orchestra.run.smithi168.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-11-29T16:16:49.180 INFO:teuthology.orchestra.run.smithi168.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-11-29T16:16:49.180 INFO:teuthology.orchestra.run.smithi168.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-11-29T16:16:49.180 INFO:teuthology.orchestra.run.smithi168.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-11-29T16:16:49.181 INFO:teuthology.orchestra.run.smithi168.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-11-29T16:16:49.181 INFO:teuthology.orchestra.run.smithi168.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-11-29T16:16:49.181 INFO:teuthology.orchestra.run.smithi168.stdout: "otp_pool": "remove.rgw.otp", 2020-11-29T16:16:49.181 INFO:teuthology.orchestra.run.smithi168.stdout: "system_key": { 2020-11-29T16:16:49.181 INFO:teuthology.orchestra.run.smithi168.stdout: "access_key": "", 2020-11-29T16:16:49.182 INFO:teuthology.orchestra.run.smithi168.stdout: "secret_key": "" 2020-11-29T16:16:49.182 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.182 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_pools": [ 2020-11-29T16:16:49.182 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.183 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "default-placement", 2020-11-29T16:16:49.183 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:16:49.183 INFO:teuthology.orchestra.run.smithi168.stdout: "index_pool": "remove.rgw.buckets.index", 2020-11-29T16:16:49.184 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": { 2020-11-29T16:16:49.184 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD": { 2020-11-29T16:16:49.184 INFO:teuthology.orchestra.run.smithi168.stdout: "data_pool": "remove.rgw.buckets.data" 2020-11-29T16:16:49.184 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.184 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.185 INFO:teuthology.orchestra.run.smithi168.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-11-29T16:16:49.185 INFO:teuthology.orchestra.run.smithi168.stdout: "index_type": 0 2020-11-29T16:16:49.185 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.185 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.185 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.186 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:49.186 INFO:teuthology.orchestra.run.smithi168.stdout: "notif_pool": "remove.rgw.log:notif" 2020-11-29T16:16:49.186 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:16:49.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.189 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:49.275 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:49.276 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:49.617 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:16:49.617 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:49.618 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 8, 2020-11-29T16:16:49.618 INFO:teuthology.orchestra.run.smithi168.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:16:49.618 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": [ 2020-11-29T16:16:49.619 INFO:teuthology.orchestra.run.smithi168.stdout: "", 2020-11-29T16:16:49.619 INFO:teuthology.orchestra.run.smithi168.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:16:49.619 INFO:teuthology.orchestra.run.smithi168.stdout: "", 2020-11-29T16:16:49.619 INFO:teuthology.orchestra.run.smithi168.stdout: "" 2020-11-29T16:16:49.619 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.620 INFO:teuthology.orchestra.run.smithi168.stdout: "period_map": { 2020-11-29T16:16:49.620 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:49.620 INFO:teuthology.orchestra.run.smithi168.stdout: "zonegroups": [ 2020-11-29T16:16:49.620 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.620 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "5494521b-4bb4-4524-99ab-1276129abed5", 2020-11-29T16:16:49.621 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "remove_zg", 2020-11-29T16:16:49.621 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "remove_zg", 2020-11-29T16:16:49.622 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "false", 2020-11-29T16:16:49.622 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [], 2020-11-29T16:16:49.622 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T16:16:49.623 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T16:16:49.623 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "9e93b063-604d-4129-bc49-ebead4c467c6", 2020-11-29T16:16:49.623 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [ 2020-11-29T16:16:49.623 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.624 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "9e93b063-604d-4129-bc49-ebead4c467c6", 2020-11-29T16:16:49.624 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "remove", 2020-11-29T16:16:49.624 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [], 2020-11-29T16:16:49.624 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:49.624 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "false", 2020-11-29T16:16:49.625 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:49.625 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:49.625 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:49.625 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:49.625 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:49.626 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:49.626 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.626 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.626 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [ 2020-11-29T16:16:49.627 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.627 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default-placement", 2020-11-29T16:16:49.627 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-11-29T16:16:49.627 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": [ 2020-11-29T16:16:49.627 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD" 2020-11-29T16:16:49.628 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:49.628 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.628 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.628 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "default-placement", 2020-11-29T16:16:49.629 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:49.629 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T16:16:49.629 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T16:16:49.629 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.630 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.630 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.630 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "test-zonegroup", 2020-11-29T16:16:49.630 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zonegroup", 2020-11-29T16:16:49.630 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "", 2020-11-29T16:16:49.631 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "true", 2020-11-29T16:16:49.631 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:49.631 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:49.631 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.631 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T16:16:49.632 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T16:16:49.632 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:49.632 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [ 2020-11-29T16:16:49.632 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.632 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:49.633 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone3", 2020-11-29T16:16:49.633 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:49.633 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:16:49.633 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.633 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:49.634 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:49.634 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:49.634 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:49.635 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:49.635 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:49.635 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:49.636 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:49.636 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.637 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.637 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:49.637 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone2", 2020-11-29T16:16:49.637 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:49.637 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:16:49.638 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.638 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:49.638 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:49.638 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:49.639 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:49.639 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:49.639 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:49.639 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:49.640 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:49.640 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.640 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.640 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:49.641 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone4", 2020-11-29T16:16:49.641 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:49.641 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:16:49.641 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.641 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:49.642 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:49.642 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:49.642 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:49.665 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "pubsub", 2020-11-29T16:16:49.665 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "false", 2020-11-29T16:16:49.665 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [ 2020-11-29T16:16:49.666 INFO:teuthology.orchestra.run.smithi168.stdout: "test-zone1" 2020-11-29T16:16:49.666 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.667 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:49.667 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.667 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.668 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:49.668 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T16:16:49.669 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:49.669 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:49.669 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.669 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:49.670 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:49.670 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:49.671 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:49.671 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:49.671 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:49.671 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:49.671 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:49.672 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.672 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.672 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [ 2020-11-29T16:16:49.673 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.673 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default-placement", 2020-11-29T16:16:49.673 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-11-29T16:16:49.673 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": [ 2020-11-29T16:16:49.674 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD" 2020-11-29T16:16:49.674 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:49.674 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.675 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.675 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "default-placement", 2020-11-29T16:16:49.675 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:49.675 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T16:16:49.676 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T16:16:49.676 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.676 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.677 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:49.677 INFO:teuthology.orchestra.run.smithi168.stdout: "short_zone_ids": [ 2020-11-29T16:16:49.677 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.678 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:49.678 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 791374490 2020-11-29T16:16:49.679 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.679 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.679 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:49.680 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 2491292681 2020-11-29T16:16:49.680 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.681 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.681 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:49.681 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 293306859 2020-11-29T16:16:49.682 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.682 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.683 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "9e93b063-604d-4129-bc49-ebead4c467c6", 2020-11-29T16:16:49.683 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 99400784 2020-11-29T16:16:49.683 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.684 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:49.684 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:49.685 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 1785095687 2020-11-29T16:16:49.685 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.685 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:49.685 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.686 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:16:49.686 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:49.686 INFO:teuthology.orchestra.run.smithi168.stdout: "period_config": { 2020-11-29T16:16:49.686 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_quota": { 2020-11-29T16:16:49.687 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T16:16:49.687 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T16:16:49.687 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T16:16:49.687 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T16:16:49.687 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T16:16:49.688 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.688 INFO:teuthology.orchestra.run.smithi168.stdout: "user_quota": { 2020-11-29T16:16:49.688 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T16:16:49.688 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T16:16:49.688 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T16:16:49.689 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T16:16:49.689 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T16:16:49.689 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:49.689 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:49.689 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:49.690 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_name": "test-realm", 2020-11-29T16:16:49.690 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:16:49.690 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:16:49.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.692 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:49.767 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:49.768 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:49.823 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:16:49.813+0000 7f9a14a16080 0 zone id 9e93b063-604d-4129-bc49-ebead4c467c6 is not a part of zonegroup test-zonegroup 2020-11-29T16:16:49.825 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:16:49.817+0000 7f9a14a16080 0 zone id 9e93b063-604d-4129-bc49-ebead4c467c6 is not a part of zonegroup default 2020-11-29T16:16:49.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.844 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:49.924 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:49.924 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:49.971 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:16:49.961+0000 7f77b387e080 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-11-29T16:16:49.979 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-11-29T16:16:49.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:49.981 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:50.057 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:50.057 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:50.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:50.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:16:50.098 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:16:50.176 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:50.177 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:50.544 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:16:50.545 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:50.545 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 9, 2020-11-29T16:16:50.545 INFO:teuthology.orchestra.run.smithi168.stdout: "predecessor_uuid": "f053a404-d473-420f-9a01-38eedc856256", 2020-11-29T16:16:50.545 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": [ 2020-11-29T16:16:50.546 INFO:teuthology.orchestra.run.smithi168.stdout: "", 2020-11-29T16:16:50.546 INFO:teuthology.orchestra.run.smithi168.stdout: "1_1606666200.930041_148.1", 2020-11-29T16:16:50.546 INFO:teuthology.orchestra.run.smithi168.stdout: "", 2020-11-29T16:16:50.547 INFO:teuthology.orchestra.run.smithi168.stdout: "" 2020-11-29T16:16:50.547 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.547 INFO:teuthology.orchestra.run.smithi168.stdout: "period_map": { 2020-11-29T16:16:50.547 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:50.548 INFO:teuthology.orchestra.run.smithi168.stdout: "zonegroups": [ 2020-11-29T16:16:50.548 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.548 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "test-zonegroup", 2020-11-29T16:16:50.548 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zonegroup", 2020-11-29T16:16:50.548 INFO:teuthology.orchestra.run.smithi168.stdout: "api_name": "", 2020-11-29T16:16:50.549 INFO:teuthology.orchestra.run.smithi168.stdout: "is_master": "true", 2020-11-29T16:16:50.549 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:50.549 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:50.549 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.550 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames": [], 2020-11-29T16:16:50.550 INFO:teuthology.orchestra.run.smithi168.stdout: "hostnames_s3website": [], 2020-11-29T16:16:50.550 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:50.551 INFO:teuthology.orchestra.run.smithi168.stdout: "zones": [ 2020-11-29T16:16:50.551 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.551 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:50.551 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone3", 2020-11-29T16:16:50.552 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:50.552 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8001" 2020-11-29T16:16:50.552 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.553 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:50.553 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:50.553 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:50.553 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:50.553 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:50.554 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:50.554 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:50.554 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:50.554 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.555 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.555 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:50.555 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone2", 2020-11-29T16:16:50.555 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:50.556 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-11-29T16:16:50.556 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.556 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:50.556 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:50.557 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:50.557 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:50.557 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:50.557 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:50.557 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:50.558 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:50.558 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.558 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.558 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:50.558 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone4", 2020-11-29T16:16:50.559 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:50.559 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-11-29T16:16:50.559 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.560 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:50.560 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:50.560 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:50.560 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:50.561 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "pubsub", 2020-11-29T16:16:50.561 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "false", 2020-11-29T16:16:50.561 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [ 2020-11-29T16:16:50.562 INFO:teuthology.orchestra.run.smithi168.stdout: "test-zone1" 2020-11-29T16:16:50.562 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.562 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:50.563 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.563 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.563 INFO:teuthology.orchestra.run.smithi168.stdout: "id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:50.564 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "test-zone1", 2020-11-29T16:16:50.564 INFO:teuthology.orchestra.run.smithi168.stdout: "endpoints": [ 2020-11-29T16:16:50.565 INFO:teuthology.orchestra.run.smithi168.stdout: "http://smithi168.front.sepia.ceph.com:8000" 2020-11-29T16:16:50.565 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.565 INFO:teuthology.orchestra.run.smithi168.stdout: "log_meta": "false", 2020-11-29T16:16:50.565 INFO:teuthology.orchestra.run.smithi168.stdout: "log_data": "true", 2020-11-29T16:16:50.566 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_index_max_shards": 11, 2020-11-29T16:16:50.566 INFO:teuthology.orchestra.run.smithi168.stdout: "read_only": "false", 2020-11-29T16:16:50.566 INFO:teuthology.orchestra.run.smithi168.stdout: "tier_type": "", 2020-11-29T16:16:50.566 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from_all": "true", 2020-11-29T16:16:50.566 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_from": [], 2020-11-29T16:16:50.567 INFO:teuthology.orchestra.run.smithi168.stdout: "redirect_zone": "" 2020-11-29T16:16:50.567 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:50.567 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.567 INFO:teuthology.orchestra.run.smithi168.stdout: "placement_targets": [ 2020-11-29T16:16:50.567 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.568 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "default-placement", 2020-11-29T16:16:50.568 INFO:teuthology.orchestra.run.smithi168.stdout: "tags": [], 2020-11-29T16:16:50.568 INFO:teuthology.orchestra.run.smithi168.stdout: "storage_classes": [ 2020-11-29T16:16:50.568 INFO:teuthology.orchestra.run.smithi168.stdout: "STANDARD" 2020-11-29T16:16:50.569 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:50.569 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:50.569 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.570 INFO:teuthology.orchestra.run.smithi168.stdout: "default_placement": "default-placement", 2020-11-29T16:16:50.570 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:50.570 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_policy": { 2020-11-29T16:16:50.571 INFO:teuthology.orchestra.run.smithi168.stdout: "groups": [] 2020-11-29T16:16:50.571 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:50.571 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:50.571 INFO:teuthology.orchestra.run.smithi168.stdout: ], 2020-11-29T16:16:50.572 INFO:teuthology.orchestra.run.smithi168.stdout: "short_zone_ids": [ 2020-11-29T16:16:50.572 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.573 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:16:50.573 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 791374490 2020-11-29T16:16:50.573 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.573 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.574 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:16:50.574 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 2491292681 2020-11-29T16:16:50.574 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.575 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.575 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "82b01792-db79-401e-8298-c419db80f2a2", 2020-11-29T16:16:50.575 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 293306859 2020-11-29T16:16:50.575 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.575 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:50.576 INFO:teuthology.orchestra.run.smithi168.stdout: "key": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:50.576 INFO:teuthology.orchestra.run.smithi168.stdout: "val": 1785095687 2020-11-29T16:16:50.576 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:50.576 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:16:50.576 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.577 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zonegroup": "test-zonegroup", 2020-11-29T16:16:50.577 INFO:teuthology.orchestra.run.smithi168.stdout: "master_zone": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:16:50.577 INFO:teuthology.orchestra.run.smithi168.stdout: "period_config": { 2020-11-29T16:16:50.577 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_quota": { 2020-11-29T16:16:50.577 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T16:16:50.578 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T16:16:50.578 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T16:16:50.578 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T16:16:50.578 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T16:16:50.579 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.579 INFO:teuthology.orchestra.run.smithi168.stdout: "user_quota": { 2020-11-29T16:16:50.579 INFO:teuthology.orchestra.run.smithi168.stdout: "enabled": false, 2020-11-29T16:16:50.579 INFO:teuthology.orchestra.run.smithi168.stdout: "check_on_raw": false, 2020-11-29T16:16:50.579 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size": -1, 2020-11-29T16:16:50.580 INFO:teuthology.orchestra.run.smithi168.stdout: "max_size_kb": 0, 2020-11-29T16:16:50.580 INFO:teuthology.orchestra.run.smithi168.stdout: "max_objects": -1 2020-11-29T16:16:50.580 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:50.580 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:50.581 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_id": "a2a72fe6-27cf-4286-bdf9-88ee6023d0e2", 2020-11-29T16:16:50.581 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_name": "test-realm", 2020-11-29T16:16:50.581 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:16:50.582 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:16:50.582 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-11-29T16:16:50.583 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-50 2020-11-29T16:16:51.322 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-51 2020-11-29T16:16:51.463 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-52 2020-11-29T16:16:51.541 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-53 2020-11-29T16:16:51.597 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-54 2020-11-29T16:16:51.679 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-55 2020-11-29T16:16:53.954 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-11-29T16:16:53.955 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-56 2020-11-29T16:16:53.998 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-57 2020-11-29T16:16:54.068 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-58 2020-11-29T16:16:54.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:54.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:54.138 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:16:54.174 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:54.174 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:54.387 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:16:54.417 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:16:54.417 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:54.418 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:16:54.418 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:16:54.418 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:54.418 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:54.418 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666614.108822_370.1", 2020-11-29T16:16:54.419 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:16:54.108822Z" 2020-11-29T16:16:54.419 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:54.419 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:54.419 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:16:54.420 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:16:54.420 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:16:54.420 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:16:54.420 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:16:54.420 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:16:54.421 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:16:54.421 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:16:54.422 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:16:54.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:54.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:54.423 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:16:54.461 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:54.461 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:16:54.732 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:16:54.733 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:16:54.733 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:16:54.733 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:16:54.733 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:16:54.734 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:16:54.734 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:16:54.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:54.734 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:16:54.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:54.735 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:16:54.735 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:54.735 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:54.735 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:16:54.736 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:54.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:16:54.736 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:54.736 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:16:54.737 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:54.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:54.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:54.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:54.737 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:16:54.738 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:54.738 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:54.738 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:16:54.738 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:54.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:54.739 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:54.739 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:16:54.739 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:16:54.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:54.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:54.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:54.740 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:16:54.741 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:54.741 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:54.741 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:16:54.741 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:54.741 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:54.742 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:54.742 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:16:54.742 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:54.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:54.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:54.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:16:54.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:16:54.743 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:16:54.744 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:16:54.744 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:16:54.744 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:16:54.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:16:54.745 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:16:54.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:16:54.745 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:16:54.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:54.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:54.746 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:16:54.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:16:54.746 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:16:54.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:16:54.747 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:16:54.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:16:54.747 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:16:54.748 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:16:59.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:59.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:16:59.752 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:16:59.793 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:16:59.794 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:00.051 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:00.051 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:00.052 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:00.052 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:00.052 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:00.052 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:00.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:00.053 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:00.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:00.053 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:00.053 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:00.054 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:00.054 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:00.054 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:00.054 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:00.055 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:00.055 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:00.055 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:00.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:00.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:00.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:00.056 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:00.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:00.056 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:00.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:00.057 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:00.057 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:00.057 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:00.057 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:00.058 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:00.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:00.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:00.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:00.058 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:00.059 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:00.059 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:00.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:00.059 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:00.059 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:00.060 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:00.060 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:00.060 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:00.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:00.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:00.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:00.061 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:00.061 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:00.061 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:00.062 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:00.062 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:00.062 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:00.062 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:00.062 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:00.063 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:00.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:00.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:00.063 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:00.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:00.064 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:00.064 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:00.064 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:00.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:00.064 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:00.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:05.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:05.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:05.068 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:05.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:05.109 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:05.392 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:05.393 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:05.393 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:05.393 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:05.394 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:05.394 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:05.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:05.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:05.395 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:05.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:05.395 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:05.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:05.395 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:05.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:05.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:05.396 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:05.396 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:05.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:05.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:05.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:05.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:05.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:05.398 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:05.398 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:05.398 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:05.398 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:05.399 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:05.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:05.399 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:05.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:05.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:05.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:05.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:05.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:05.400 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:05.401 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:05.401 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:05.401 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:05.401 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:05.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:05.406 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:05.407 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:10.409 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:10.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:10.410 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:10.452 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:10.453 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:10.726 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:10.727 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:10.727 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:10.727 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:10.728 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:10.728 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:10.728 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:10.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:10.728 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:10.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:10.729 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:10.729 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:10.729 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:10.730 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:10.730 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:10.730 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:10.730 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:10.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:10.740 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:10.740 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:10.740 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:10.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:10.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:10.741 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:10.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:10.742 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:10.742 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:10.742 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:10.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:10.742 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:10.744 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:15.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:15.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:15.747 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:15.790 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:15.790 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:16.060 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:16.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:16.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:16.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:16.073 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:16.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:16.073 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:16.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:16.073 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:16.075 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:21.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:21.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:21.077 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:21.117 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:21.118 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:21.578 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:21.579 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:21.579 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:21.579 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:21.580 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:21.580 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:21.580 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:21.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:21.580 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:21.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:21.581 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:21.581 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:21.581 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:21.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:21.582 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:21.582 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:21.582 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:21.582 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:21.583 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:21.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:21.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:21.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:21.583 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:21.584 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:21.584 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:21.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:21.584 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:21.585 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:21.585 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:21.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:21.585 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:21.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:21.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:21.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:21.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:21.587 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:21.587 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:21.587 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:21.587 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:21.588 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:21.588 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:21.588 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:21.588 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:21.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:21.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:21.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:21.589 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:21.589 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:21.589 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:21.590 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:21.590 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:21.590 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:21.590 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:21.591 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:21.592 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:21.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:21.593 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:21.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:26.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:26.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:26.597 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:26.636 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:26.637 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:26.924 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:26.924 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:26.925 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:26.925 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:26.925 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:26.925 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:26.926 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:26.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:26.926 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:26.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:26.927 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:26.927 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:26.927 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:26.927 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:26.927 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:26.928 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:26.928 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:26.928 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:26.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:26.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:26.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:26.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:26.929 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:26.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:26.930 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:26.930 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:26.930 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:26.930 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:26.930 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:26.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:26.931 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:26.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:26.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:26.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:26.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:26.932 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:26.933 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:26.933 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:26.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:26.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:26.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:26.934 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:26.934 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:26.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:26.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:26.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:26.936 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:26.936 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:26.937 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:26.937 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:26.937 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:26.938 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:26.938 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:26.939 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:26.939 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:26.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:26.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:26.940 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:26.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:26.942 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:26.942 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:26.942 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:26.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:26.943 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:26.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:31.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:31.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:31.947 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:31.987 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:31.988 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:32.272 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:32.272 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:32.272 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:32.273 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:32.273 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:32.273 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:32.273 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:32.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:32.274 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:32.275 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:32.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:32.275 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:32.275 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:32.275 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:32.276 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:32.276 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:32.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:32.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:32.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:32.277 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:32.277 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:32.277 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:32.277 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:32.278 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:32.278 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:32.278 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:32.278 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:32.278 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:32.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:32.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:32.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:32.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:32.279 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:32.280 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:32.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:32.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:32.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:32.280 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:32.281 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:32.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:32.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:32.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:32.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:32.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:32.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:32.282 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:32.282 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:32.282 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:32.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:32.283 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:32.283 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:32.283 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:32.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:32.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:32.284 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:32.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:32.284 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:32.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:32.285 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:32.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:32.285 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:32.286 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:37.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:37.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:37.289 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:37.329 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:37.329 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:37.604 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:37.604 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:37.605 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:37.605 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:37.605 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:37.605 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:37.606 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:37.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:37.606 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:37.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:37.606 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:37.607 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:37.607 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:37.607 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:37.607 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:37.607 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:37.608 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:37.608 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:37.608 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:37.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:37.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:37.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:37.609 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:37.609 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:37.609 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:37.609 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:37.610 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:37.610 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:37.610 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:37.610 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:37.611 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:37.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:37.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:37.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:37.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:37.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:37.612 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:37.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:37.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:37.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:37.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:37.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:37.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:37.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:37.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:37.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:37.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:37.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:37.614 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:37.614 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:37.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:37.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:37.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:37.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:37.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:37.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:37.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:37.616 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:37.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:37.616 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:37.617 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:37.617 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:37.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:37.617 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:37.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:42.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:42.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:42.620 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:42.660 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:42.661 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:42.934 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:42.935 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:42.935 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:42.935 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:42.936 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:42.936 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:42.937 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:42.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:42.937 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:42.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:42.938 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:42.939 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:42.939 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:42.940 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:42.940 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:42.940 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:42.941 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:42.941 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:42.942 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:42.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:42.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:42.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:42.943 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:42.943 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:42.944 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:42.944 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:42.944 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:42.945 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:42.945 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:42.945 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:42.945 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:42.946 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:42.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:42.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:42.946 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:42.947 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:42.947 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:42.947 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:42.947 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:42.947 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:42.948 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:42.948 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:42.948 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:42.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:42.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:42.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:42.949 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:42.949 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:42.949 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:42.950 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:42.950 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:42.950 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:42.950 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:42.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:42.951 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:42.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:42.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:42.951 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:42.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:42.952 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:42.952 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:42.952 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:42.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:42.953 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:42.954 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:47.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:47.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:47.957 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:47.997 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:47.998 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:48.267 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:48.268 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:48.268 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:48.268 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:48.268 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:48.269 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:48.269 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:48.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:48.269 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:48.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:48.270 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:48.270 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:48.270 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:48.270 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:48.271 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:48.271 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:48.271 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:48.271 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:48.271 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:48.272 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:48.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:48.272 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:48.272 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:48.272 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:48.273 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:48.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:48.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:48.273 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:48.273 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:48.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:48.274 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:48.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:48.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:48.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:48.275 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:48.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:48.275 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:48.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:48.276 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:48.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:48.276 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:48.276 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:48.276 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:48.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:48.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:48.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:48.277 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:48.277 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:48.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:48.281 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:48.282 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:53.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:53.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:53.284 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:53.324 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:53.325 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:53.603 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:53.603 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:53.606 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:53.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:53.606 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:53.607 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:53.607 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:53.608 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:53.608 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:53.608 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:53.609 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:53.609 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:53.609 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:53.609 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:53.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:53.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:53.610 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:53.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:53.611 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:53.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:53.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:53.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:53.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:53.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:53.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:53.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:53.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:53.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:53.615 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:53.615 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:53.615 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:53.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:53.616 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:53.616 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:53.616 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:53.617 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:53.617 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:53.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:53.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:53.618 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:53.619 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:53.619 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:53.619 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:53.619 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:53.620 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:53.620 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:53.620 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:53.623 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:53.625 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:17:58.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:58.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:17:58.628 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:17:58.670 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:17:58.670 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:17:58.957 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:17:58.957 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:17:58.957 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:17:58.958 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:17:58.958 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:17:58.958 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:17:58.958 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:58.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:58.959 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:17:58.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:58.959 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:17:58.960 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:58.960 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:58.960 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:17:58.960 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:58.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:17:58.961 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:58.961 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:17:58.961 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:58.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:58.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:58.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:58.962 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:17:58.962 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:58.962 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:58.963 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:17:58.963 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:58.963 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:58.963 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:17:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:17:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:58.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:17:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:58.965 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:58.966 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:17:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:17:58.967 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:17:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:17:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:17:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:17:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:17:58.968 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:17:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:17:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:17:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:58.969 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:17:58.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:17:58.970 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:17:58.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:17:58.970 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:17:58.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:17:58.971 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:17:58.972 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:03.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:03.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:03.974 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:04.015 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:04.016 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:04.297 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:04.298 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:04.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:04.300 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:04.300 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:04.300 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:04.300 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:04.300 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:04.301 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:04.301 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:04.301 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:04.301 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:04.301 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:04.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:04.302 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:04.302 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:04.302 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:04.303 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:04.303 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:04.303 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:04.303 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:04.303 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:04.304 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:04.304 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:04.304 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:04.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:04.304 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:04.305 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:04.305 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:04.305 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:04.305 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:04.305 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:04.306 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:04.306 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:04.306 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:04.306 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:04.306 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:04.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:04.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:04.307 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:04.307 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:04.307 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:04.308 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:04.308 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:04.308 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:04.308 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:04.308 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:04.309 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:04.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:04.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:04.309 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:04.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:04.310 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:04.310 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:04.310 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:04.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:04.310 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:04.311 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:09.313 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:09.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:09.314 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:09.355 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:09.355 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:09.638 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:09.638 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:09.639 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:09.639 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:09.639 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:09.639 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:09.649 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:09.651 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:09.652 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:09.652 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:09.652 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:09.652 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:09.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:09.653 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:09.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:09.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:09.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:09.654 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:09.654 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:09.654 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:09.654 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:09.655 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:09.655 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:09.655 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:09.655 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:09.655 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:09.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:09.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:09.656 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:09.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:09.656 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:09.657 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:09.657 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:09.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:09.657 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:09.658 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:14.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:14.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:14.661 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:14.701 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:14.701 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:14.981 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:14.981 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:14.981 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:14.982 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:14.982 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:14.982 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:14.982 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:14.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:14.983 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:14.983 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:14.983 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:14.984 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:14.984 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:14.984 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:14.984 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:14.984 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:14.985 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:14.985 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:14.985 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:14.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:14.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:14.986 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:14.986 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:14.986 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:14.986 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:14.987 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:14.987 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:14.987 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:14.987 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:14.987 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:14.988 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:14.988 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:14.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:14.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:14.989 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:14.989 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:14.989 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:14.989 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:14.989 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:14.990 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:14.990 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:14.990 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:14.990 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:14.990 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:14.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:14.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:14.991 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:14.991 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:14.991 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:14.992 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:14.992 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:14.992 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:14.992 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:14.993 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:14.993 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:14.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:14.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:14.993 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:14.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:14.994 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:14.994 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:14.994 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:14.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:14.995 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:14.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:19.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:19.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:19.998 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:20.041 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:20.042 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:20.310 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:20.311 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:20.312 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:20.313 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:20.314 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:20.315 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:20.315 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:20.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:20.315 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:20.315 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:20.316 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:20.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:20.318 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:20.319 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:20.320 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:20.321 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:20.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:20.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:20.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:20.324 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:20.325 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:25.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:25.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:25.328 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:25.368 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:25.369 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:25.636 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:25.636 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:25.636 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:25.636 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:25.637 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:25.637 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:25.637 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:25.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:25.638 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:25.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:25.638 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:25.639 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:25.639 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:25.639 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:25.639 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:25.639 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:25.640 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:25.640 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:25.640 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:25.640 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:25.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:25.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:25.641 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:25.641 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:25.641 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:25.642 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:25.642 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:25.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:25.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:25.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:25.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:25.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:25.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:25.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:25.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:25.644 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:25.644 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:25.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:25.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:25.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:25.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:25.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:25.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:25.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:25.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:25.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:25.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:25.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:25.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:25.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:25.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:25.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:25.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:25.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:25.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:25.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:25.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:25.649 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:25.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:25.649 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:25.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:25.650 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:25.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:25.650 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:25.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:30.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:30.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:30.654 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:30.694 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:30.694 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:30.997 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:30.997 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:30.998 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:30.998 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:30.998 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:30.998 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:30.999 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:30.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:30.999 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:30.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:30.999 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:31.000 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:31.000 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:31.000 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:31.000 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:31.001 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:31.001 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:31.001 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:31.001 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:31.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:31.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:31.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:31.003 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:31.003 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:31.003 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:31.004 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:31.004 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:31.004 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:31.005 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:31.005 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:31.005 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:31.006 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:31.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:31.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:31.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:31.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:31.007 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:31.007 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:31.007 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:31.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:31.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:31.008 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:31.008 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:31.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:31.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:31.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:31.009 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:31.009 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:31.009 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:31.009 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:31.010 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:31.010 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:31.010 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:31.010 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:31.011 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:31.011 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:31.011 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:31.011 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:31.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:31.012 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:31.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:31.012 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:31.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:31.012 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:31.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:36.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:36.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:36.016 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:36.056 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:36.056 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:36.330 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:36.330 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:36.331 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:36.331 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:36.331 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:36.331 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:36.331 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:36.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:36.332 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:36.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:36.332 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:36.332 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:36.333 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:36.333 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:36.333 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:36.333 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:36.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:36.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:36.334 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:36.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:36.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:36.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:36.335 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:36.335 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:36.335 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:36.335 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:36.336 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:36.336 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:36.336 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:36.336 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:36.336 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:36.337 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:36.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:36.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:36.337 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:36.337 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:36.338 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:36.338 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:36.338 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:36.338 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:36.338 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:36.339 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:36.339 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:36.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:36.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:36.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:36.340 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:36.340 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:36.340 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:36.340 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:36.340 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:36.341 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:36.341 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:36.341 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:36.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:36.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:36.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:36.342 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:36.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:36.342 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:36.343 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:36.343 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:36.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:36.343 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:36.344 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:41.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:41.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:41.346 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:41.389 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:41.389 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:41.668 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:41.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:41.679 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:41.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:41.681 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:41.682 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:41.682 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:41.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:46.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:46.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:46.685 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:46.727 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:46.727 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:47.003 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:47.004 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:47.004 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:47.004 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:47.004 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:47.004 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:47.005 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:47.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:47.005 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:47.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:47.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:47.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:47.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:47.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:47.007 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:47.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:47.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:47.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:47.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:47.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:47.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:47.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:47.008 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:47.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:47.010 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:47.010 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:47.010 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:47.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:47.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:47.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:47.011 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:47.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:47.011 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:47.012 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:47.012 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:47.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:47.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:47.013 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:47.013 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:47.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:47.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:47.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:47.014 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:47.014 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:47.014 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:47.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:47.015 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:47.015 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:47.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:47.015 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:47.016 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:47.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:47.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:47.016 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:47.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:47.017 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:47.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:47.017 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:47.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:47.017 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:47.019 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:52.020 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:52.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:52.022 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:52.064 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:52.064 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:52.333 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:52.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:52.336 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:52.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:52.336 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:52.336 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:52.337 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:52.337 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:52.337 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:52.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:52.338 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:52.338 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:52.338 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:52.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:52.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666351.336452_248.1", 2020-11-29T16:18:52.340 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:52.340 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:52.341 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:52.341 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:12:31.336452Z", 2020-11-29T16:18:52.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:52.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:52.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:52.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:52.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:52.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:52.343 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:52.343 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:52.343 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:52.343 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:52.343 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:52.349 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:52.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666614.108822_370.1 target=1_1606666351.336452_248.1 2020-11-29T16:18:57.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:57.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:57.353 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:57.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:57.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:57.677 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:57.677 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:57.678 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:57.678 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:57.678 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:57.678 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:57.679 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:57.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:57.679 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:57.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:57.679 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:57.680 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:57.680 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:57.680 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:57.680 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:57.681 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:57.681 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:57.681 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:57.681 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:57.681 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:57.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:57.682 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:57.682 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:57.682 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:57.683 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:57.683 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666614.108822_370.1", 2020-11-29T16:18:57.683 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:57.683 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:57.683 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:57.684 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:16:54.108822Z", 2020-11-29T16:18:57.684 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:57.684 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:57.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:57.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:57.685 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:57.685 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:57.685 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:57.685 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:57.686 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:57.686 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:57.686 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:57.686 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:57.686 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:57.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:57.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:57.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:57.687 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:57.687 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:57.688 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:57.688 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:57.688 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:57.688 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:57.688 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:57.689 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:57.689 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:57.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:57.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:57.689 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:57.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:57.690 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:57.690 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:57.690 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:57.691 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:57.691 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:57.692 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:18:57.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:18:57.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:57.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:57.694 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:57.736 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:57.736 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:58.013 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:18:58.014 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:18:58.014 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:18:58.014 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:18:58.015 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:18:58.015 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:58.015 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:18:58.015 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:18:58.016 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:18:58.016 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:18:58.016 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:18:58.016 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:18:58.017 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:18:58.017 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:58.017 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:18:58.017 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:18:58.017 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:18:58.018 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:58.018 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:18:58.018 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:18:58.018 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:18:58.019 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:18:58.019 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:18:58.019 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:18:58.019 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:18:58.020 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666614.108822_370.1", 2020-11-29T16:18:58.020 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:18:58.020 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:18:58.020 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:18:58.020 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:16:54.108822Z", 2020-11-29T16:18:58.021 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:18:58.021 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:18:58.021 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:18:58.021 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:18:58.022 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:18:58.022 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:18:58.022 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:18:58.022 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:18:58.023 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:18:58.023 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:18:58.023 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:18:58.023 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:18:58.023 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:18:58.024 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:18:58.024 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:18:58.024 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:18:58.024 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:18:58.025 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:18:58.025 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:18:58.025 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:18:58.025 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:18:58.025 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:18:58.026 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:18:58.026 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:18:58.026 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:18:58.026 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:18:58.027 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:18:58.027 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:18:58.027 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:18:58.027 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:18:58.027 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:18:58.028 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:18:58.028 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:18:58.028 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:18:58.029 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:18:58.030 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:18:58.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:58.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:58.031 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:58.072 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:58.072 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:58.443 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:18:58.444 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:18:58.444 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:18:58.444 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:18:58.444 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:18:58.445 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:18:58.445 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:58.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:58.445 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:18:58.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:58.446 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:18:58.446 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:58.446 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:58.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:18:58.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:58.447 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:18:58.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:58.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:18:58.448 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:58.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:58.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:58.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:58.449 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:18:58.449 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:58.449 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:58.449 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666614.108822_370.1", 2020-11-29T16:18:58.450 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:58.450 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:58.450 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:58.451 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:16:54.108822Z", 2020-11-29T16:18:58.451 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:18:58.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:58.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:58.451 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:58.452 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:18:58.452 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:58.452 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:58.452 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:58.453 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:58.453 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:58.453 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:58.453 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:58.453 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:58.454 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:58.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:58.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:18:58.454 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:18:58.455 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:18:58.455 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:18:58.455 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:18:58.455 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:18:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:18:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:18:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:18:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:18:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:18:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:18:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:18:58.458 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:18:58.458 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:18:58.458 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:18:58.458 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:18:58.459 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:18:58.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:18:58.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:18:58.461 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:18:58.504 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:58.504 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:58.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:18:58.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:18:58.874 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:18:58.956 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:58.957 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:59.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:18:59.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:18:59.314 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:18:59.394 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:59.395 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:18:59.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:59.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:18:59.748 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:18:59.826 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:18:59.827 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:00.141 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:19:00.142 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000078", 2020-11-29T16:19:00.142 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.history", 2020-11-29T16:19:00.142 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000070", 2020-11-29T16:19:00.142 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000104", 2020-11-29T16:19:00.143 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:7", 2020-11-29T16:19:00.143 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4-kcmyep-47", 2020-11-29T16:19:00.143 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000026", 2020-11-29T16:19:00.143 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.36679b47-164f-436b-b728-5846b32cea58.1", 2020-11-29T16:19:00.144 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.3c165286-3738-4e0f-b399-acd859302998.1", 2020-11-29T16:19:00.144 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000028", 2020-11-29T16:19:00.144 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36:10", 2020-11-29T16:19:00.144 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4-kcmyep-4", 2020-11-29T16:19:00.144 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000040", 2020-11-29T16:19:00.145 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.3.Dzu6K7qv+Vn.0", 2020-11-29T16:19:00.145 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.full-sync.index.0", 2020-11-29T16:19:00.145 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26-kcmyep-26", 2020-11-29T16:19:00.145 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000015", 2020-11-29T16:19:00.145 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3-kcmyep-3", 2020-11-29T16:19:00.146 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000069", 2020-11-29T16:19:00.146 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000095", 2020-11-29T16:19:00.146 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000003", 2020-11-29T16:19:00.146 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000047", 2020-11-29T16:19:00.146 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-46:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3:8", 2020-11-29T16:19:00.147 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29-kcmyep-29", 2020-11-29T16:19:00.147 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.0.Z/h54yAMi3k.0", 2020-11-29T16:19:00.147 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000035", 2020-11-29T16:19:00.147 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000037", 2020-11-29T16:19:00.148 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21:8", 2020-11-29T16:19:00.148 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15:1", 2020-11-29T16:19:00.148 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000024", 2020-11-29T16:19:00.148 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.7-kcmyep-56", 2020-11-29T16:19:00.148 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.36679b47-164f-436b-b728-5846b32cea58.3", 2020-11-29T16:19:00.149 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000000", 2020-11-29T16:19:00.149 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000031", 2020-11-29T16:19:00.149 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.3", 2020-11-29T16:19:00.149 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:7", 2020-11-29T16:19:00.149 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24:8", 2020-11-29T16:19:00.150 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21-kcmyep-21", 2020-11-29T16:19:00.150 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000076", 2020-11-29T16:19:00.150 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000113", 2020-11-29T16:19:00.150 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000005", 2020-11-29T16:19:00.150 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000011", 2020-11-29T16:19:00.151 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000058", 2020-11-29T16:19:00.151 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000052", 2020-11-29T16:19:00.151 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25-kcmyep-25", 2020-11-29T16:19:00.151 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:3", 2020-11-29T16:19:00.151 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:3", 2020-11-29T16:19:00.152 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40-kcmyep-40", 2020-11-29T16:19:00.152 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:3", 2020-11-29T16:19:00.152 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36-kcmyep-36", 2020-11-29T16:19:00.152 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000088", 2020-11-29T16:19:00.153 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1-kcmyep-44", 2020-11-29T16:19:00.153 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:3", 2020-11-29T16:19:00.153 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000080", 2020-11-29T16:19:00.153 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000090", 2020-11-29T16:19:00.153 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14-kcmyep-14", 2020-11-29T16:19:00.154 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000110", 2020-11-29T16:19:00.154 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000096", 2020-11-29T16:19:00.154 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38:10", 2020-11-29T16:19:00.154 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.3c165286-3738-4e0f-b399-acd859302998.1", 2020-11-29T16:19:00.154 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000087", 2020-11-29T16:19:00.155 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000008", 2020-11-29T16:19:00.155 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.36679b47-164f-436b-b728-5846b32cea58.2", 2020-11-29T16:19:00.155 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:19:00.155 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.4-kcmyep-53", 2020-11-29T16:19:00.155 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:1", 2020-11-29T16:19:00.156 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000006", 2020-11-29T16:19:00.156 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000029", 2020-11-29T16:19:00.156 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-3c165286-3738-4e0f-b399-acd859302998.5116.1-kcmyep-42", 2020-11-29T16:19:00.156 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:3", 2020-11-29T16:19:00.156 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000089", 2020-11-29T16:19:00.157 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000022", 2020-11-29T16:19:00.157 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39-kcmyep-39", 2020-11-29T16:19:00.157 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36:10", 2020-11-29T16:19:00.157 INFO:teuthology.orchestra.run.smithi168.stdout: "bilog.trim", 2020-11-29T16:19:00.157 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000016", 2020-11-29T16:19:00.158 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000041", 2020-11-29T16:19:00.158 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000018", 2020-11-29T16:19:00.158 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000092", 2020-11-29T16:19:00.158 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.27-kcmyep-27", 2020-11-29T16:19:00.159 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.36679b47-164f-436b-b728-5846b32cea58.3", 2020-11-29T16:19:00.159 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:5", 2020-11-29T16:19:00.159 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000014", 2020-11-29T16:19:00.159 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37:10", 2020-11-29T16:19:00.159 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26:8", 2020-11-29T16:19:00.160 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28-kcmyep-28", 2020-11-29T16:19:00.160 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24:8", 2020-11-29T16:19:00.160 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21:8", 2020-11-29T16:19:00.160 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000112", 2020-11-29T16:19:00.160 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000007", 2020-11-29T16:19:00.161 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000021", 2020-11-29T16:19:00.161 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:1", 2020-11-29T16:19:00.161 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000064", 2020-11-29T16:19:00.161 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.3c165286-3738-4e0f-b399-acd859302998.3", 2020-11-29T16:19:00.161 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000071", 2020-11-29T16:19:00.162 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12-kcmyep-12", 2020-11-29T16:19:00.162 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000074", 2020-11-29T16:19:00.162 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:5", 2020-11-29T16:19:00.162 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22-kcmyep-22", 2020-11-29T16:19:00.162 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000081", 2020-11-29T16:19:00.163 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20:8", 2020-11-29T16:19:00.163 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000009", 2020-11-29T16:19:00.163 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.3c165286-3738-4e0f-b399-acd859302998.0", 2020-11-29T16:19:00.163 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000121", 2020-11-29T16:19:00.164 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.1", 2020-11-29T16:19:00.164 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:8", 2020-11-29T16:19:00.164 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000125", 2020-11-29T16:19:00.164 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000082", 2020-11-29T16:19:00.164 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.full-sync.index.1", 2020-11-29T16:19:00.165 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23:8", 2020-11-29T16:19:00.165 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10-kcmyep-10", 2020-11-29T16:19:00.165 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20-kcmyep-20", 2020-11-29T16:19:00.165 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000105", 2020-11-29T16:19:00.165 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000059", 2020-11-29T16:19:00.166 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2-kcmyep-2", 2020-11-29T16:19:00.166 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000077", 2020-11-29T16:19:00.166 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-48:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5:8", 2020-11-29T16:19:00.166 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000032", 2020-11-29T16:19:00.166 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:3", 2020-11-29T16:19:00.167 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15:1", 2020-11-29T16:19:00.167 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.5-kcmyep-54", 2020-11-29T16:19:00.167 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000053", 2020-11-29T16:19:00.167 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:5", 2020-11-29T16:19:00.167 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000091", 2020-11-29T16:19:00.168 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24-kcmyep-24", 2020-11-29T16:19:00.168 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000065", 2020-11-29T16:19:00.168 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000083", 2020-11-29T16:19:00.168 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000010", 2020-11-29T16:19:00.169 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:8", 2020-11-29T16:19:00.169 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status.shard.2", 2020-11-29T16:19:00.169 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:8", 2020-11-29T16:19:00.169 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15-kcmyep-15", 2020-11-29T16:19:00.169 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:8", 2020-11-29T16:19:00.170 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3-kcmyep-46", 2020-11-29T16:19:00.170 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000045", 2020-11-29T16:19:00.170 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000002", 2020-11-29T16:19:00.170 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000116", 2020-11-29T16:19:00.170 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000034", 2020-11-29T16:19:00.171 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000101", 2020-11-29T16:19:00.171 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.36679b47-164f-436b-b728-5846b32cea58.0", 2020-11-29T16:19:00.171 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status.shard.1", 2020-11-29T16:19:00.171 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13-kcmyep-13", 2020-11-29T16:19:00.171 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:1", 2020-11-29T16:19:00.172 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22:8", 2020-11-29T16:19:00.172 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000079", 2020-11-29T16:19:00.172 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000049", 2020-11-29T16:19:00.172 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1-kcmyep-49", 2020-11-29T16:19:00.172 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.log.24836445-da90-4fa5-864b-eda958e6bd14.1", 2020-11-29T16:19:00.173 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000117", 2020-11-29T16:19:00.173 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000044", 2020-11-29T16:19:00.173 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:3", 2020-11-29T16:19:00.173 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11-kcmyep-11", 2020-11-29T16:19:00.173 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23-kcmyep-23", 2020-11-29T16:19:00.174 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000066", 2020-11-29T16:19:00.174 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:19:00.174 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:1", 2020-11-29T16:19:00.174 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000068", 2020-11-29T16:19:00.175 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8-kcmyep-8", 2020-11-29T16:19:00.175 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000085", 2020-11-29T16:19:00.175 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000073", 2020-11-29T16:19:00.175 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:3", 2020-11-29T16:19:00.175 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000038", 2020-11-29T16:19:00.176 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.full-sync.index.3", 2020-11-29T16:19:00.176 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000118", 2020-11-29T16:19:00.176 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.3c165286-3738-4e0f-b399-acd859302998.2", 2020-11-29T16:19:00.176 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20:8", 2020-11-29T16:19:00.176 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000036", 2020-11-29T16:19:00.177 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:3", 2020-11-29T16:19:00.177 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000103", 2020-11-29T16:19:00.177 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status.shard.3", 2020-11-29T16:19:00.177 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.1.N5fzZwYU1Ty.0", 2020-11-29T16:19:00.177 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000119", 2020-11-29T16:19:00.178 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.30-kcmyep-30", 2020-11-29T16:19:00.178 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000098", 2020-11-29T16:19:00.178 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22:8", 2020-11-29T16:19:00.178 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status.shard.0", 2020-11-29T16:19:00.178 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:1", 2020-11-29T16:19:00.179 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000027", 2020-11-29T16:19:00.179 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2-kcmyep-45", 2020-11-29T16:19:00.179 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status", 2020-11-29T16:19:00.179 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000019", 2020-11-29T16:19:00.179 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.33-kcmyep-33", 2020-11-29T16:19:00.180 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000039", 2020-11-29T16:19:00.180 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:7", 2020-11-29T16:19:00.180 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:1", 2020-11-29T16:19:00.180 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.1-kcmyep-50", 2020-11-29T16:19:00.180 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000100", 2020-11-29T16:19:00.181 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18-kcmyep-18", 2020-11-29T16:19:00.181 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5-kcmyep-5", 2020-11-29T16:19:00.181 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000093", 2020-11-29T16:19:00.181 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32-kcmyep-32", 2020-11-29T16:19:00.181 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13:1", 2020-11-29T16:19:00.182 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000004", 2020-11-29T16:19:00.182 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000063", 2020-11-29T16:19:00.182 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000122", 2020-11-29T16:19:00.182 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:1", 2020-11-29T16:19:00.183 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000057", 2020-11-29T16:19:00.183 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000054", 2020-11-29T16:19:00.183 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000114", 2020-11-29T16:19:00.183 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.0", 2020-11-29T16:19:00.183 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-48:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5:8", 2020-11-29T16:19:00.184 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:7", 2020-11-29T16:19:00.184 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19-kcmyep-19", 2020-11-29T16:19:00.184 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000012", 2020-11-29T16:19:00.184 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000084", 2020-11-29T16:19:00.184 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000043", 2020-11-29T16:19:00.185 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34-kcmyep-34", 2020-11-29T16:19:00.185 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.3c165286-3738-4e0f-b399-acd859302998.0", 2020-11-29T16:19:00.185 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38-kcmyep-38", 2020-11-29T16:19:00.185 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13:1", 2020-11-29T16:19:00.185 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000111", 2020-11-29T16:19:00.186 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:8", 2020-11-29T16:19:00.186 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000048", 2020-11-29T16:19:00.186 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37-kcmyep-37", 2020-11-29T16:19:00.186 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000020", 2020-11-29T16:19:00.186 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:5", 2020-11-29T16:19:00.187 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000099", 2020-11-29T16:19:00.187 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000056", 2020-11-29T16:19:00.187 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:1", 2020-11-29T16:19:00.187 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.36679b47-164f-436b-b728-5846b32cea58.1", 2020-11-29T16:19:00.187 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:3", 2020-11-29T16:19:00.188 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:3", 2020-11-29T16:19:00.188 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000072", 2020-11-29T16:19:00.188 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000062", 2020-11-29T16:19:00.188 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000115", 2020-11-29T16:19:00.189 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000033", 2020-11-29T16:19:00.189 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:8", 2020-11-29T16:19:00.189 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:5", 2020-11-29T16:19:00.189 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.6-kcmyep-55", 2020-11-29T16:19:00.189 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000025", 2020-11-29T16:19:00.190 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000123", 2020-11-29T16:19:00.190 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000108", 2020-11-29T16:19:00.190 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000094", 2020-11-29T16:19:00.190 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-47:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4:8", 2020-11-29T16:19:00.190 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1-kcmyep-1", 2020-11-29T16:19:00.191 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-49:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1:5", 2020-11-29T16:19:00.191 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14:1", 2020-11-29T16:19:00.191 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:3", 2020-11-29T16:19:00.191 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000060", 2020-11-29T16:19:00.191 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000109", 2020-11-29T16:19:00.192 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:8", 2020-11-29T16:19:00.192 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41-kcmyep-41", 2020-11-29T16:19:00.192 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:1", 2020-11-29T16:19:00.192 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-3c165286-3738-4e0f-b399-acd859302998.5116.2-kcmyep-43", 2020-11-29T16:19:00.192 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:8", 2020-11-29T16:19:00.193 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000013", 2020-11-29T16:19:00.193 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.9-kcmyep-58", 2020-11-29T16:19:00.193 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5-kcmyep-48", 2020-11-29T16:19:00.193 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000051", 2020-11-29T16:19:00.194 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000106", 2020-11-29T16:19:00.194 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.2-kcmyep-51", 2020-11-29T16:19:00.194 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000017", 2020-11-29T16:19:00.194 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000023", 2020-11-29T16:19:00.194 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000107", 2020-11-29T16:19:00.195 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000046", 2020-11-29T16:19:00.195 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:1", 2020-11-29T16:19:00.195 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38:10", 2020-11-29T16:19:00.195 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.full-sync.index.2", 2020-11-29T16:19:00.195 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-47:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4:8", 2020-11-29T16:19:00.196 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-46:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3:8", 2020-11-29T16:19:00.196 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:7", 2020-11-29T16:19:00.196 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:1", 2020-11-29T16:19:00.196 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-49:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1:5", 2020-11-29T16:19:00.196 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.2", 2020-11-29T16:19:00.197 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000067", 2020-11-29T16:19:00.197 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:7", 2020-11-29T16:19:00.197 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7-kcmyep-7", 2020-11-29T16:19:00.197 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000086", 2020-11-29T16:19:00.198 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37:10", 2020-11-29T16:19:00.198 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14:1", 2020-11-29T16:19:00.198 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:5", 2020-11-29T16:19:00.198 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.3c165286-3738-4e0f-b399-acd859302998.3", 2020-11-29T16:19:00.198 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35-kcmyep-35", 2020-11-29T16:19:00.199 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.3c165286-3738-4e0f-b399-acd859302998.2", 2020-11-29T16:19:00.199 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000050", 2020-11-29T16:19:00.199 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.2.WRFpVBlZHxU.0", 2020-11-29T16:19:00.199 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000102", 2020-11-29T16:19:00.199 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:5", 2020-11-29T16:19:00.200 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9-kcmyep-9", 2020-11-29T16:19:00.200 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25:8", 2020-11-29T16:19:00.200 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000124", 2020-11-29T16:19:00.200 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:5", 2020-11-29T16:19:00.200 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17-kcmyep-17", 2020-11-29T16:19:00.201 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.36679b47-164f-436b-b728-5846b32cea58.2", 2020-11-29T16:19:00.201 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31-kcmyep-31", 2020-11-29T16:19:00.201 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000001", 2020-11-29T16:19:00.201 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16-kcmyep-16", 2020-11-29T16:19:00.202 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000126", 2020-11-29T16:19:00.202 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26:8", 2020-11-29T16:19:00.202 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000055", 2020-11-29T16:19:00.202 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.36679b47-164f-436b-b728-5846b32cea58.0", 2020-11-29T16:19:00.202 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000061", 2020-11-29T16:19:00.203 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23:8", 2020-11-29T16:19:00.203 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000097", 2020-11-29T16:19:00.203 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000042", 2020-11-29T16:19:00.203 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6-kcmyep-6", 2020-11-29T16:19:00.203 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.8-kcmyep-57", 2020-11-29T16:19:00.204 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000075", 2020-11-29T16:19:00.204 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000120", 2020-11-29T16:19:00.204 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000030", 2020-11-29T16:19:00.204 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:1", 2020-11-29T16:19:00.204 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.3-kcmyep-52", 2020-11-29T16:19:00.205 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25:8" 2020-11-29T16:19:00.205 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:19:00.205 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T16:19:00.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:00.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:00.208 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:00.245 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:00.245 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:00.548 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:19:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000078", 2020-11-29T16:19:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.history", 2020-11-29T16:19:00.548 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000070", 2020-11-29T16:19:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000104", 2020-11-29T16:19:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:7", 2020-11-29T16:19:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4-kcmyep-47", 2020-11-29T16:19:00.549 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000026", 2020-11-29T16:19:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.36679b47-164f-436b-b728-5846b32cea58.1", 2020-11-29T16:19:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14:1", 2020-11-29T16:19:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000028", 2020-11-29T16:19:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4-kcmyep-4", 2020-11-29T16:19:00.550 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15:1", 2020-11-29T16:19:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000040", 2020-11-29T16:19:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.0", 2020-11-29T16:19:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26-kcmyep-26", 2020-11-29T16:19:00.551 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000015", 2020-11-29T16:19:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3-kcmyep-3", 2020-11-29T16:19:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000069", 2020-11-29T16:19:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000095", 2020-11-29T16:19:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000003", 2020-11-29T16:19:00.552 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000047", 2020-11-29T16:19:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000035", 2020-11-29T16:19:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000037", 2020-11-29T16:19:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:3", 2020-11-29T16:19:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25:8", 2020-11-29T16:19:00.553 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:8", 2020-11-29T16:19:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21:8", 2020-11-29T16:19:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000024", 2020-11-29T16:19:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.36679b47-164f-436b-b728-5846b32cea58.3", 2020-11-29T16:19:00.554 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000000", 2020-11-29T16:19:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000031", 2020-11-29T16:19:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3", 2020-11-29T16:19:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24:8", 2020-11-29T16:19:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:1", 2020-11-29T16:19:00.555 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:7", 2020-11-29T16:19:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:3", 2020-11-29T16:19:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-46:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3:8", 2020-11-29T16:19:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21-kcmyep-21", 2020-11-29T16:19:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.2", 2020-11-29T16:19:00.556 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000076", 2020-11-29T16:19:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24:8", 2020-11-29T16:19:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000113", 2020-11-29T16:19:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000005", 2020-11-29T16:19:00.557 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21:8", 2020-11-29T16:19:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000011", 2020-11-29T16:19:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000058", 2020-11-29T16:19:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000052", 2020-11-29T16:19:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25-kcmyep-25", 2020-11-29T16:19:00.558 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.0", 2020-11-29T16:19:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:3", 2020-11-29T16:19:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40-kcmyep-40", 2020-11-29T16:19:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36-kcmyep-36", 2020-11-29T16:19:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000088", 2020-11-29T16:19:00.559 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1-kcmyep-44", 2020-11-29T16:19:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000080", 2020-11-29T16:19:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000090", 2020-11-29T16:19:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14-kcmyep-14", 2020-11-29T16:19:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000110", 2020-11-29T16:19:00.560 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000096", 2020-11-29T16:19:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:3", 2020-11-29T16:19:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000087", 2020-11-29T16:19:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000008", 2020-11-29T16:19:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.36679b47-164f-436b-b728-5846b32cea58.2", 2020-11-29T16:19:00.561 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.36679b47-164f-436b-b728-5846b32cea58", 2020-11-29T16:19:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:1", 2020-11-29T16:19:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000006", 2020-11-29T16:19:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000029", 2020-11-29T16:19:00.562 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-3c165286-3738-4e0f-b399-acd859302998.5116.1-kcmyep-42", 2020-11-29T16:19:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:3", 2020-11-29T16:19:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000089", 2020-11-29T16:19:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000022", 2020-11-29T16:19:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39-kcmyep-39", 2020-11-29T16:19:00.563 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36:10", 2020-11-29T16:19:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "bilog.trim", 2020-11-29T16:19:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000016", 2020-11-29T16:19:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000041", 2020-11-29T16:19:00.564 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3.EXQ7qV/OXTz.0", 2020-11-29T16:19:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000018", 2020-11-29T16:19:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000092", 2020-11-29T16:19:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.36679b47-164f-436b-b728-5846b32cea58.3", 2020-11-29T16:19:00.565 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:5", 2020-11-29T16:19:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.2", 2020-11-29T16:19:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22:8", 2020-11-29T16:19:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000014", 2020-11-29T16:19:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37:10", 2020-11-29T16:19:00.566 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.28-kcmyep-28", 2020-11-29T16:19:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36:10", 2020-11-29T16:19:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000112", 2020-11-29T16:19:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000007", 2020-11-29T16:19:00.567 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000021", 2020-11-29T16:19:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:1", 2020-11-29T16:19:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000064", 2020-11-29T16:19:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000071", 2020-11-29T16:19:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12-kcmyep-12", 2020-11-29T16:19:00.568 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000074", 2020-11-29T16:19:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:19:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.1", 2020-11-29T16:19:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:5", 2020-11-29T16:19:00.569 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22-kcmyep-22", 2020-11-29T16:19:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000081", 2020-11-29T16:19:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20:8", 2020-11-29T16:19:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000009", 2020-11-29T16:19:00.570 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000121", 2020-11-29T16:19:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1", 2020-11-29T16:19:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000125", 2020-11-29T16:19:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000082", 2020-11-29T16:19:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.1", 2020-11-29T16:19:00.571 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23:8", 2020-11-29T16:19:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:5", 2020-11-29T16:19:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10-kcmyep-10", 2020-11-29T16:19:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20-kcmyep-20", 2020-11-29T16:19:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000105", 2020-11-29T16:19:00.572 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000059", 2020-11-29T16:19:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2-kcmyep-2", 2020-11-29T16:19:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000077", 2020-11-29T16:19:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-48:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5:8", 2020-11-29T16:19:00.573 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-48:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5:8", 2020-11-29T16:19:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:1", 2020-11-29T16:19:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000032", 2020-11-29T16:19:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:3", 2020-11-29T16:19:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15:1", 2020-11-29T16:19:00.574 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.5-kcmyep-54", 2020-11-29T16:19:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:3", 2020-11-29T16:19:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000053", 2020-11-29T16:19:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:5", 2020-11-29T16:19:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000091", 2020-11-29T16:19:00.575 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24-kcmyep-24", 2020-11-29T16:19:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000065", 2020-11-29T16:19:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000083", 2020-11-29T16:19:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000010", 2020-11-29T16:19:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:8", 2020-11-29T16:19:00.576 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.2", 2020-11-29T16:19:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:8", 2020-11-29T16:19:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15-kcmyep-15", 2020-11-29T16:19:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:8", 2020-11-29T16:19:00.577 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3-kcmyep-46", 2020-11-29T16:19:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000045", 2020-11-29T16:19:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-49:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1:5", 2020-11-29T16:19:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000002", 2020-11-29T16:19:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000116", 2020-11-29T16:19:00.578 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.3", 2020-11-29T16:19:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000034", 2020-11-29T16:19:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000101", 2020-11-29T16:19:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.36679b47-164f-436b-b728-5846b32cea58.0", 2020-11-29T16:19:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.1", 2020-11-29T16:19:00.579 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13-kcmyep-13", 2020-11-29T16:19:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:1", 2020-11-29T16:19:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:1", 2020-11-29T16:19:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000079", 2020-11-29T16:19:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000049", 2020-11-29T16:19:00.580 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1-kcmyep-49", 2020-11-29T16:19:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.24836445-da90-4fa5-864b-eda958e6bd14.1", 2020-11-29T16:19:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000117", 2020-11-29T16:19:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000044", 2020-11-29T16:19:00.581 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:3", 2020-11-29T16:19:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:7", 2020-11-29T16:19:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11-kcmyep-11", 2020-11-29T16:19:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23-kcmyep-23", 2020-11-29T16:19:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000066", 2020-11-29T16:19:00.582 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:1", 2020-11-29T16:19:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000068", 2020-11-29T16:19:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8-kcmyep-8", 2020-11-29T16:19:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000085", 2020-11-29T16:19:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000073", 2020-11-29T16:19:00.583 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000038", 2020-11-29T16:19:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.3", 2020-11-29T16:19:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000118", 2020-11-29T16:19:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000036", 2020-11-29T16:19:00.584 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000103", 2020-11-29T16:19:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.3", 2020-11-29T16:19:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000119", 2020-11-29T16:19:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000098", 2020-11-29T16:19:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22:8", 2020-11-29T16:19:00.585 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.0", 2020-11-29T16:19:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000027", 2020-11-29T16:19:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2-kcmyep-45", 2020-11-29T16:19:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status", 2020-11-29T16:19:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000019", 2020-11-29T16:19:00.586 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1.eCRdzOw5ews.0", 2020-11-29T16:19:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000039", 2020-11-29T16:19:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:7", 2020-11-29T16:19:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:7", 2020-11-29T16:19:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:1", 2020-11-29T16:19:00.587 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000100", 2020-11-29T16:19:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23:8", 2020-11-29T16:19:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.1", 2020-11-29T16:19:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18-kcmyep-18", 2020-11-29T16:19:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5-kcmyep-5", 2020-11-29T16:19:00.588 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13:1", 2020-11-29T16:19:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000093", 2020-11-29T16:19:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000004", 2020-11-29T16:19:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000063", 2020-11-29T16:19:00.589 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000122", 2020-11-29T16:19:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-47:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4:8", 2020-11-29T16:19:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:1", 2020-11-29T16:19:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000057", 2020-11-29T16:19:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:3", 2020-11-29T16:19:00.590 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:1", 2020-11-29T16:19:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000054", 2020-11-29T16:19:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000114", 2020-11-29T16:19:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0", 2020-11-29T16:19:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:7", 2020-11-29T16:19:00.591 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19-kcmyep-19", 2020-11-29T16:19:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000012", 2020-11-29T16:19:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000084", 2020-11-29T16:19:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000043", 2020-11-29T16:19:00.592 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.34-kcmyep-34", 2020-11-29T16:19:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38-kcmyep-38", 2020-11-29T16:19:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13:1", 2020-11-29T16:19:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000111", 2020-11-29T16:19:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:8", 2020-11-29T16:19:00.593 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000048", 2020-11-29T16:19:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37-kcmyep-37", 2020-11-29T16:19:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000020", 2020-11-29T16:19:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:5", 2020-11-29T16:19:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20:8", 2020-11-29T16:19:00.594 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0.imDfnkxB5az.0", 2020-11-29T16:19:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000099", 2020-11-29T16:19:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000056", 2020-11-29T16:19:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.36679b47-164f-436b-b728-5846b32cea58.1", 2020-11-29T16:19:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:1", 2020-11-29T16:19:00.595 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:3", 2020-11-29T16:19:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000072", 2020-11-29T16:19:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:3", 2020-11-29T16:19:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000062", 2020-11-29T16:19:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2.0jquE/djwwC.0", 2020-11-29T16:19:00.596 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000115", 2020-11-29T16:19:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000033", 2020-11-29T16:19:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:5", 2020-11-29T16:19:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000025", 2020-11-29T16:19:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000123", 2020-11-29T16:19:00.597 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000108", 2020-11-29T16:19:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:8", 2020-11-29T16:19:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000094", 2020-11-29T16:19:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1-kcmyep-1", 2020-11-29T16:19:00.598 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-49:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1:5", 2020-11-29T16:19:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:3", 2020-11-29T16:19:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000060", 2020-11-29T16:19:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000109", 2020-11-29T16:19:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41-kcmyep-41", 2020-11-29T16:19:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-3c165286-3738-4e0f-b399-acd859302998.5116.2-kcmyep-43", 2020-11-29T16:19:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000013", 2020-11-29T16:19:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26:8", 2020-11-29T16:19:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38:10", 2020-11-29T16:19:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5-kcmyep-48", 2020-11-29T16:19:00.600 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000051", 2020-11-29T16:19:00.601 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37:10", 2020-11-29T16:19:00.601 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000106", 2020-11-29T16:19:00.601 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.2-kcmyep-51", 2020-11-29T16:19:00.601 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000017", 2020-11-29T16:19:00.601 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000023", 2020-11-29T16:19:00.602 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:1", 2020-11-29T16:19:00.602 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000107", 2020-11-29T16:19:00.602 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000046", 2020-11-29T16:19:00.602 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38:10", 2020-11-29T16:19:00.602 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.2", 2020-11-29T16:19:00.603 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-47:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4:8", 2020-11-29T16:19:00.603 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-46:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3:8", 2020-11-29T16:19:00.603 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2", 2020-11-29T16:19:00.603 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000067", 2020-11-29T16:19:00.604 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.3", 2020-11-29T16:19:00.604 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7-kcmyep-7", 2020-11-29T16:19:00.604 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000086", 2020-11-29T16:19:00.604 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14:1", 2020-11-29T16:19:00.604 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:5", 2020-11-29T16:19:00.605 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000050", 2020-11-29T16:19:00.605 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000102", 2020-11-29T16:19:00.605 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9-kcmyep-9", 2020-11-29T16:19:00.605 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:5", 2020-11-29T16:19:00.605 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000124", 2020-11-29T16:19:00.606 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17-kcmyep-17", 2020-11-29T16:19:00.606 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.36679b47-164f-436b-b728-5846b32cea58.2", 2020-11-29T16:19:00.606 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.31-kcmyep-31", 2020-11-29T16:19:00.606 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000001", 2020-11-29T16:19:00.606 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16-kcmyep-16", 2020-11-29T16:19:00.607 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000126", 2020-11-29T16:19:00.607 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26:8", 2020-11-29T16:19:00.607 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000055", 2020-11-29T16:19:00.607 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.36679b47-164f-436b-b728-5846b32cea58.0", 2020-11-29T16:19:00.607 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000061", 2020-11-29T16:19:00.608 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:8", 2020-11-29T16:19:00.608 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000097", 2020-11-29T16:19:00.608 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:8", 2020-11-29T16:19:00.608 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.0", 2020-11-29T16:19:00.608 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000042", 2020-11-29T16:19:00.609 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6-kcmyep-6", 2020-11-29T16:19:00.609 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.8-kcmyep-57", 2020-11-29T16:19:00.609 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000075", 2020-11-29T16:19:00.609 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000120", 2020-11-29T16:19:00.609 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000030", 2020-11-29T16:19:00.610 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.36679b47-164f-436b-b728-5846b32cea58:kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25:8" 2020-11-29T16:19:00.610 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:19:00.610 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T16:19:00.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:00.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:00.612 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:00.649 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:00.650 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:00.924 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:19:00.925 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000078", 2020-11-29T16:19:00.925 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.history", 2020-11-29T16:19:00.925 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000070", 2020-11-29T16:19:00.926 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000104", 2020-11-29T16:19:00.926 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4-kcmyep-47", 2020-11-29T16:19:00.926 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000026", 2020-11-29T16:19:00.926 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14:1", 2020-11-29T16:19:00.927 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.3c165286-3738-4e0f-b399-acd859302998.1", 2020-11-29T16:19:00.927 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000028", 2020-11-29T16:19:00.927 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36:10", 2020-11-29T16:19:00.927 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.4-kcmyep-4", 2020-11-29T16:19:00.927 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15:1", 2020-11-29T16:19:00.928 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000040", 2020-11-29T16:19:00.928 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.full-sync.index.0", 2020-11-29T16:19:00.928 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26-kcmyep-26", 2020-11-29T16:19:00.928 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000015", 2020-11-29T16:19:00.929 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.3-kcmyep-3", 2020-11-29T16:19:00.929 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000069", 2020-11-29T16:19:00.929 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000095", 2020-11-29T16:19:00.929 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000003", 2020-11-29T16:19:00.929 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000047", 2020-11-29T16:19:00.930 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-46:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3:8", 2020-11-29T16:19:00.930 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.29-kcmyep-29", 2020-11-29T16:19:00.930 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000035", 2020-11-29T16:19:00.930 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000037", 2020-11-29T16:19:00.930 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:3", 2020-11-29T16:19:00.931 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25:8", 2020-11-29T16:19:00.931 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:8", 2020-11-29T16:19:00.931 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15:1", 2020-11-29T16:19:00.931 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000024", 2020-11-29T16:19:00.932 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000000", 2020-11-29T16:19:00.932 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000031", 2020-11-29T16:19:00.932 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.3", 2020-11-29T16:19:00.932 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:7", 2020-11-29T16:19:00.932 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:1", 2020-11-29T16:19:00.933 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:7", 2020-11-29T16:19:00.933 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:3", 2020-11-29T16:19:00.933 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-46:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3:8", 2020-11-29T16:19:00.933 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21-kcmyep-21", 2020-11-29T16:19:00.934 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.2", 2020-11-29T16:19:00.934 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000076", 2020-11-29T16:19:00.934 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24:8", 2020-11-29T16:19:00.934 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000113", 2020-11-29T16:19:00.935 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000005", 2020-11-29T16:19:00.935 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21:8", 2020-11-29T16:19:00.935 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000011", 2020-11-29T16:19:00.935 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000058", 2020-11-29T16:19:00.936 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000052", 2020-11-29T16:19:00.936 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25-kcmyep-25", 2020-11-29T16:19:00.936 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.0", 2020-11-29T16:19:00.937 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:3", 2020-11-29T16:19:00.937 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.40-kcmyep-40", 2020-11-29T16:19:00.937 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:3", 2020-11-29T16:19:00.937 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36-kcmyep-36", 2020-11-29T16:19:00.937 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000088", 2020-11-29T16:19:00.938 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.1-kcmyep-44", 2020-11-29T16:19:00.938 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:3", 2020-11-29T16:19:00.938 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000080", 2020-11-29T16:19:00.939 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000090", 2020-11-29T16:19:00.939 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14-kcmyep-14", 2020-11-29T16:19:00.939 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000110", 2020-11-29T16:19:00.939 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000096", 2020-11-29T16:19:00.939 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38:10", 2020-11-29T16:19:00.940 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:3", 2020-11-29T16:19:00.940 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.3c165286-3738-4e0f-b399-acd859302998.1", 2020-11-29T16:19:00.940 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000087", 2020-11-29T16:19:00.940 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000008", 2020-11-29T16:19:00.940 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000006", 2020-11-29T16:19:00.941 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000029", 2020-11-29T16:19:00.941 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-3c165286-3738-4e0f-b399-acd859302998.5116.1-kcmyep-42", 2020-11-29T16:19:00.941 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000089", 2020-11-29T16:19:00.941 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000022", 2020-11-29T16:19:00.942 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.39-kcmyep-39", 2020-11-29T16:19:00.942 INFO:teuthology.orchestra.run.smithi168.stdout: "bilog.trim", 2020-11-29T16:19:00.942 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000016", 2020-11-29T16:19:00.942 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000041", 2020-11-29T16:19:00.943 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000018", 2020-11-29T16:19:00.943 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000092", 2020-11-29T16:19:00.943 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:5", 2020-11-29T16:19:00.943 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.0.juq+szdUH0L.0", 2020-11-29T16:19:00.943 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:5", 2020-11-29T16:19:00.944 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.2", 2020-11-29T16:19:00.944 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22:8", 2020-11-29T16:19:00.944 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000014", 2020-11-29T16:19:00.944 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26:8", 2020-11-29T16:19:00.944 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36:10", 2020-11-29T16:19:00.945 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24:8", 2020-11-29T16:19:00.945 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21:8", 2020-11-29T16:19:00.945 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000112", 2020-11-29T16:19:00.945 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000007", 2020-11-29T16:19:00.945 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000021", 2020-11-29T16:19:00.946 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000064", 2020-11-29T16:19:00.946 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.3c165286-3738-4e0f-b399-acd859302998.3", 2020-11-29T16:19:00.946 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000071", 2020-11-29T16:19:00.946 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12-kcmyep-12", 2020-11-29T16:19:00.947 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000074", 2020-11-29T16:19:00.947 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:19:00.947 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.1", 2020-11-29T16:19:00.947 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22-kcmyep-22", 2020-11-29T16:19:00.947 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000081", 2020-11-29T16:19:00.948 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000009", 2020-11-29T16:19:00.948 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.3c165286-3738-4e0f-b399-acd859302998.0", 2020-11-29T16:19:00.948 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000121", 2020-11-29T16:19:00.948 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.1", 2020-11-29T16:19:00.949 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:8", 2020-11-29T16:19:00.949 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000125", 2020-11-29T16:19:00.949 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000082", 2020-11-29T16:19:00.949 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.full-sync.index.1", 2020-11-29T16:19:00.949 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:5", 2020-11-29T16:19:00.950 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10-kcmyep-10", 2020-11-29T16:19:00.950 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20-kcmyep-20", 2020-11-29T16:19:00.950 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000105", 2020-11-29T16:19:00.950 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000059", 2020-11-29T16:19:00.950 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.2-kcmyep-2", 2020-11-29T16:19:00.951 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000077", 2020-11-29T16:19:00.951 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-48:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5:8", 2020-11-29T16:19:00.951 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:1", 2020-11-29T16:19:00.951 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000032", 2020-11-29T16:19:00.952 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:3", 2020-11-29T16:19:00.952 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000053", 2020-11-29T16:19:00.952 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000091", 2020-11-29T16:19:00.952 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24-kcmyep-24", 2020-11-29T16:19:00.952 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000065", 2020-11-29T16:19:00.953 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000083", 2020-11-29T16:19:00.953 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000010", 2020-11-29T16:19:00.953 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status.shard.2", 2020-11-29T16:19:00.953 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15-kcmyep-15", 2020-11-29T16:19:00.954 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3-kcmyep-46", 2020-11-29T16:19:00.954 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000045", 2020-11-29T16:19:00.954 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-49:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1:5", 2020-11-29T16:19:00.954 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000002", 2020-11-29T16:19:00.955 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000116", 2020-11-29T16:19:00.955 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.3", 2020-11-29T16:19:00.955 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000034", 2020-11-29T16:19:00.955 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000101", 2020-11-29T16:19:00.955 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status.shard.1", 2020-11-29T16:19:00.956 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13-kcmyep-13", 2020-11-29T16:19:00.956 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22:8", 2020-11-29T16:19:00.956 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:1", 2020-11-29T16:19:00.956 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000079", 2020-11-29T16:19:00.956 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000049", 2020-11-29T16:19:00.957 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1-kcmyep-49", 2020-11-29T16:19:00.957 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.log.24836445-da90-4fa5-864b-eda958e6bd14.1", 2020-11-29T16:19:00.957 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000117", 2020-11-29T16:19:00.957 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000044", 2020-11-29T16:19:00.957 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:7", 2020-11-29T16:19:00.958 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11-kcmyep-11", 2020-11-29T16:19:00.958 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23-kcmyep-23", 2020-11-29T16:19:00.958 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000066", 2020-11-29T16:19:00.958 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.3c165286-3738-4e0f-b399-acd859302998", 2020-11-29T16:19:00.959 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000068", 2020-11-29T16:19:00.959 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.8-kcmyep-8", 2020-11-29T16:19:00.959 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000085", 2020-11-29T16:19:00.959 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000073", 2020-11-29T16:19:00.959 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:3", 2020-11-29T16:19:00.960 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000038", 2020-11-29T16:19:00.960 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.full-sync.index.3", 2020-11-29T16:19:00.960 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000118", 2020-11-29T16:19:00.960 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.3c165286-3738-4e0f-b399-acd859302998.2", 2020-11-29T16:19:00.961 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20:8", 2020-11-29T16:19:00.961 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000036", 2020-11-29T16:19:00.961 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:3", 2020-11-29T16:19:00.961 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000103", 2020-11-29T16:19:00.961 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status.shard.3", 2020-11-29T16:19:00.962 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000119", 2020-11-29T16:19:00.962 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.log.e24cddf9-396c-4261-a7bc-b4e1e678325e.0", 2020-11-29T16:19:00.962 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000098", 2020-11-29T16:19:00.962 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status.shard.0", 2020-11-29T16:19:00.962 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:1", 2020-11-29T16:19:00.963 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000027", 2020-11-29T16:19:00.963 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.2-kcmyep-45", 2020-11-29T16:19:00.963 INFO:teuthology.orchestra.run.smithi168.stdout: "mdlog.sync-status", 2020-11-29T16:19:00.963 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000019", 2020-11-29T16:19:00.963 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000039", 2020-11-29T16:19:00.964 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:7", 2020-11-29T16:19:00.964 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000100", 2020-11-29T16:19:00.964 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23:8", 2020-11-29T16:19:00.964 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.1", 2020-11-29T16:19:00.964 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18-kcmyep-18", 2020-11-29T16:19:00.965 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.5-kcmyep-5", 2020-11-29T16:19:00.965 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13:1", 2020-11-29T16:19:00.965 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000093", 2020-11-29T16:19:00.965 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.32-kcmyep-32", 2020-11-29T16:19:00.966 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13:1", 2020-11-29T16:19:00.966 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000004", 2020-11-29T16:19:00.966 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000063", 2020-11-29T16:19:00.966 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000122", 2020-11-29T16:19:00.966 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-47:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4:8", 2020-11-29T16:19:00.967 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000057", 2020-11-29T16:19:00.967 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:3", 2020-11-29T16:19:00.967 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.3.qsEmWvjEqRA.0", 2020-11-29T16:19:00.967 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:1", 2020-11-29T16:19:00.967 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000054", 2020-11-29T16:19:00.968 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000114", 2020-11-29T16:19:00.968 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.0", 2020-11-29T16:19:00.968 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-48:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5:8", 2020-11-29T16:19:00.968 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19-kcmyep-19", 2020-11-29T16:19:00.969 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000012", 2020-11-29T16:19:00.969 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000084", 2020-11-29T16:19:00.969 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000043", 2020-11-29T16:19:00.969 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.3c165286-3738-4e0f-b399-acd859302998.0", 2020-11-29T16:19:00.970 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38-kcmyep-38", 2020-11-29T16:19:00.970 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000111", 2020-11-29T16:19:00.970 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000048", 2020-11-29T16:19:00.970 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37-kcmyep-37", 2020-11-29T16:19:00.970 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000020", 2020-11-29T16:19:00.971 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20:8", 2020-11-29T16:19:00.971 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000099", 2020-11-29T16:19:00.971 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000056", 2020-11-29T16:19:00.971 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:1", 2020-11-29T16:19:00.971 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:1", 2020-11-29T16:19:00.972 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:3", 2020-11-29T16:19:00.972 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000072", 2020-11-29T16:19:00.972 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:3", 2020-11-29T16:19:00.972 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000062", 2020-11-29T16:19:00.972 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000115", 2020-11-29T16:19:00.973 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000033", 2020-11-29T16:19:00.973 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:8", 2020-11-29T16:19:00.973 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.6-kcmyep-55", 2020-11-29T16:19:00.973 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000025", 2020-11-29T16:19:00.974 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000123", 2020-11-29T16:19:00.974 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000108", 2020-11-29T16:19:00.974 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:8", 2020-11-29T16:19:00.975 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000094", 2020-11-29T16:19:00.975 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-47:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4:8", 2020-11-29T16:19:00.975 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.1-kcmyep-1", 2020-11-29T16:19:00.975 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14:1", 2020-11-29T16:19:00.975 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000060", 2020-11-29T16:19:00.976 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000109", 2020-11-29T16:19:00.976 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:8", 2020-11-29T16:19:00.976 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.41-kcmyep-41", 2020-11-29T16:19:00.977 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:1", 2020-11-29T16:19:00.977 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-3c165286-3738-4e0f-b399-acd859302998.5116.2-kcmyep-43", 2020-11-29T16:19:00.978 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:8", 2020-11-29T16:19:00.978 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000013", 2020-11-29T16:19:00.978 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26:8", 2020-11-29T16:19:00.978 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38:10", 2020-11-29T16:19:00.979 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.9-kcmyep-58", 2020-11-29T16:19:00.979 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5-kcmyep-48", 2020-11-29T16:19:00.979 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000051", 2020-11-29T16:19:00.980 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37:10", 2020-11-29T16:19:00.980 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.2.uZOrnfkJcTX.0", 2020-11-29T16:19:00.980 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000106", 2020-11-29T16:19:00.981 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000017", 2020-11-29T16:19:00.981 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.log.f053a404-d473-420f-9a01-38eedc856256.1", 2020-11-29T16:19:00.981 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000023", 2020-11-29T16:19:00.981 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:1", 2020-11-29T16:19:00.981 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000107", 2020-11-29T16:19:00.982 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000046", 2020-11-29T16:19:00.982 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:1", 2020-11-29T16:19:00.982 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.full-sync.index.2", 2020-11-29T16:19:00.982 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:7", 2020-11-29T16:19:00.983 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:1", 2020-11-29T16:19:00.983 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-49:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1:5", 2020-11-29T16:19:00.983 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.2", 2020-11-29T16:19:00.983 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000067", 2020-11-29T16:19:00.983 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.3", 2020-11-29T16:19:00.984 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:7", 2020-11-29T16:19:00.984 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.7-kcmyep-7", 2020-11-29T16:19:00.984 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000086", 2020-11-29T16:19:00.985 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37:10", 2020-11-29T16:19:00.985 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:5", 2020-11-29T16:19:00.985 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.3c165286-3738-4e0f-b399-acd859302998.3", 2020-11-29T16:19:00.985 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.35-kcmyep-35", 2020-11-29T16:19:00.985 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:5", 2020-11-29T16:19:00.986 INFO:teuthology.orchestra.run.smithi168.stdout: "datalog.sync-status.shard.3c165286-3738-4e0f-b399-acd859302998.2", 2020-11-29T16:19:00.986 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000050", 2020-11-29T16:19:00.986 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000102", 2020-11-29T16:19:00.986 INFO:teuthology.orchestra.run.smithi168.stdout: "meta.log.e24cddf9-396c-4261-a7bc-b4e1e678325e.1", 2020-11-29T16:19:00.987 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:5", 2020-11-29T16:19:00.987 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.9-kcmyep-9", 2020-11-29T16:19:00.987 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25:8", 2020-11-29T16:19:00.987 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:5", 2020-11-29T16:19:00.988 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000124", 2020-11-29T16:19:00.988 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:5", 2020-11-29T16:19:00.988 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17-kcmyep-17", 2020-11-29T16:19:00.988 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000001", 2020-11-29T16:19:00.989 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16-kcmyep-16", 2020-11-29T16:19:00.989 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000126", 2020-11-29T16:19:00.989 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000055", 2020-11-29T16:19:00.989 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000061", 2020-11-29T16:19:00.989 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23:8", 2020-11-29T16:19:00.990 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:8", 2020-11-29T16:19:00.990 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000097", 2020-11-29T16:19:00.990 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:8", 2020-11-29T16:19:00.990 INFO:teuthology.orchestra.run.smithi168.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.0", 2020-11-29T16:19:00.991 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000042", 2020-11-29T16:19:00.991 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-15-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.6-kcmyep-6", 2020-11-29T16:19:00.991 INFO:teuthology.orchestra.run.smithi168.stdout: "data_log.1.3YaCvSZYUVV.0", 2020-11-29T16:19:00.991 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000075", 2020-11-29T16:19:00.992 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000120", 2020-11-29T16:19:00.992 INFO:teuthology.orchestra.run.smithi168.stdout: "obj_delete_at_hint.0000000030", 2020-11-29T16:19:00.992 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket.sync-status.3c165286-3738-4e0f-b399-acd859302998:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:1", 2020-11-29T16:19:00.992 INFO:teuthology.orchestra.run.smithi168.stdout: "2020-11-29-16-a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.3-kcmyep-52" 2020-11-29T16:19:00.992 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:19:00.993 INFO:teuthology.orchestra.run.smithi168.stdout: 2020-11-29T16:19:00.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:00.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:00.995 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:01.033 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:01.033 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:01.296 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:19:01.296 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000078", 2020-11-29T16:19:01.297 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.history", 2020-11-29T16:19:01.297 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000070", 2020-11-29T16:19:01.297 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000104", 2020-11-29T16:19:01.297 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1.ae+farFDmEW.0", 2020-11-29T16:19:01.298 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000026", 2020-11-29T16:19:01.298 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-14:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.14:1", 2020-11-29T16:19:01.298 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000028", 2020-11-29T16:19:01.299 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-15:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.15:1", 2020-11-29T16:19:01.299 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000040", 2020-11-29T16:19:01.299 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.0", 2020-11-29T16:19:01.299 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000015", 2020-11-29T16:19:01.300 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000069", 2020-11-29T16:19:01.300 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000095", 2020-11-29T16:19:01.300 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000003", 2020-11-29T16:19:01.301 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000047", 2020-11-29T16:19:01.301 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000035", 2020-11-29T16:19:01.302 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000037", 2020-11-29T16:19:01.302 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:3", 2020-11-29T16:19:01.302 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-25:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.25:8", 2020-11-29T16:19:01.303 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:8", 2020-11-29T16:19:01.303 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000024", 2020-11-29T16:19:01.303 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000000", 2020-11-29T16:19:01.304 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000031", 2020-11-29T16:19:01.304 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3", 2020-11-29T16:19:01.304 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:1", 2020-11-29T16:19:01.305 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:7", 2020-11-29T16:19:01.305 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:3", 2020-11-29T16:19:01.305 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-46:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.3:8", 2020-11-29T16:19:01.305 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.2", 2020-11-29T16:19:01.306 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000076", 2020-11-29T16:19:01.306 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-24:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.24:8", 2020-11-29T16:19:01.306 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000113", 2020-11-29T16:19:01.306 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000005", 2020-11-29T16:19:01.306 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-21:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.21:8", 2020-11-29T16:19:01.307 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000011", 2020-11-29T16:19:01.307 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000058", 2020-11-29T16:19:01.307 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000052", 2020-11-29T16:19:01.307 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.0", 2020-11-29T16:19:01.308 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000088", 2020-11-29T16:19:01.308 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000080", 2020-11-29T16:19:01.308 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000090", 2020-11-29T16:19:01.308 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000110", 2020-11-29T16:19:01.309 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000096", 2020-11-29T16:19:01.309 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:3", 2020-11-29T16:19:01.309 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000087", 2020-11-29T16:19:01.309 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000008", 2020-11-29T16:19:01.309 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000006", 2020-11-29T16:19:01.310 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000029", 2020-11-29T16:19:01.310 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000089", 2020-11-29T16:19:01.310 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000022", 2020-11-29T16:19:01.310 INFO:teuthology.orchestra.run.smithi184.stdout: "bilog.trim", 2020-11-29T16:19:01.311 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000016", 2020-11-29T16:19:01.311 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000041", 2020-11-29T16:19:01.311 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000018", 2020-11-29T16:19:01.311 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000092", 2020-11-29T16:19:01.312 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:5", 2020-11-29T16:19:01.312 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.2", 2020-11-29T16:19:01.312 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-22:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.22:8", 2020-11-29T16:19:01.312 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000014", 2020-11-29T16:19:01.312 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-36:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.36:10", 2020-11-29T16:19:01.313 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000112", 2020-11-29T16:19:01.313 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000007", 2020-11-29T16:19:01.313 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000021", 2020-11-29T16:19:01.313 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000064", 2020-11-29T16:19:01.313 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000071", 2020-11-29T16:19:01.314 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000074", 2020-11-29T16:19:01.314 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76", 2020-11-29T16:19:01.315 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.1", 2020-11-29T16:19:01.315 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000081", 2020-11-29T16:19:01.315 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000009", 2020-11-29T16:19:01.315 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000121", 2020-11-29T16:19:01.316 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1", 2020-11-29T16:19:01.316 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000125", 2020-11-29T16:19:01.316 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000082", 2020-11-29T16:19:01.316 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.1", 2020-11-29T16:19:01.316 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:5", 2020-11-29T16:19:01.317 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000105", 2020-11-29T16:19:01.317 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000059", 2020-11-29T16:19:01.317 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000077", 2020-11-29T16:19:01.317 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-48:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.5:8", 2020-11-29T16:19:01.317 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:1", 2020-11-29T16:19:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000032", 2020-11-29T16:19:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:3", 2020-11-29T16:19:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000053", 2020-11-29T16:19:01.318 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000091", 2020-11-29T16:19:01.319 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000065", 2020-11-29T16:19:01.319 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000083", 2020-11-29T16:19:01.319 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000010", 2020-11-29T16:19:01.319 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.2", 2020-11-29T16:19:01.319 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000045", 2020-11-29T16:19:01.320 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-49:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5351.1:5", 2020-11-29T16:19:01.320 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000002", 2020-11-29T16:19:01.320 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000116", 2020-11-29T16:19:01.320 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.3", 2020-11-29T16:19:01.320 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000034", 2020-11-29T16:19:01.321 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000101", 2020-11-29T16:19:01.321 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.1", 2020-11-29T16:19:01.321 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:1", 2020-11-29T16:19:01.321 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000079", 2020-11-29T16:19:01.322 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000049", 2020-11-29T16:19:01.322 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.24836445-da90-4fa5-864b-eda958e6bd14.1", 2020-11-29T16:19:01.322 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000117", 2020-11-29T16:19:01.322 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000044", 2020-11-29T16:19:01.322 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:7", 2020-11-29T16:19:01.323 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000066", 2020-11-29T16:19:01.323 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000068", 2020-11-29T16:19:01.323 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000085", 2020-11-29T16:19:01.323 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000073", 2020-11-29T16:19:01.323 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000038", 2020-11-29T16:19:01.324 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.3", 2020-11-29T16:19:01.324 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000118", 2020-11-29T16:19:01.324 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000036", 2020-11-29T16:19:01.324 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000103", 2020-11-29T16:19:01.324 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.3", 2020-11-29T16:19:01.325 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000119", 2020-11-29T16:19:01.325 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.e24cddf9-396c-4261-a7bc-b4e1e678325e.0", 2020-11-29T16:19:01.325 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000098", 2020-11-29T16:19:01.325 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.0", 2020-11-29T16:19:01.325 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000027", 2020-11-29T16:19:01.326 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status", 2020-11-29T16:19:01.326 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000019", 2020-11-29T16:19:01.326 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000039", 2020-11-29T16:19:01.326 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-12:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.12:7", 2020-11-29T16:19:01.327 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000100", 2020-11-29T16:19:01.327 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-23:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.23:8", 2020-11-29T16:19:01.327 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.1", 2020-11-29T16:19:01.327 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-13:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.13:1", 2020-11-29T16:19:01.327 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000093", 2020-11-29T16:19:01.328 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000004", 2020-11-29T16:19:01.328 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000063", 2020-11-29T16:19:01.328 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000122", 2020-11-29T16:19:01.329 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-47:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5152.4:8", 2020-11-29T16:19:01.329 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000057", 2020-11-29T16:19:01.329 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:3", 2020-11-29T16:19:01.330 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-17:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.17:1", 2020-11-29T16:19:01.330 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000054", 2020-11-29T16:19:01.330 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000114", 2020-11-29T16:19:01.331 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0", 2020-11-29T16:19:01.331 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000012", 2020-11-29T16:19:01.331 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000084", 2020-11-29T16:19:01.332 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000043", 2020-11-29T16:19:01.332 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000111", 2020-11-29T16:19:01.333 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000048", 2020-11-29T16:19:01.333 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000020", 2020-11-29T16:19:01.333 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-20:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.20:8", 2020-11-29T16:19:01.333 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000099", 2020-11-29T16:19:01.334 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000056", 2020-11-29T16:19:01.334 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:1", 2020-11-29T16:19:01.334 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000072", 2020-11-29T16:19:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:3", 2020-11-29T16:19:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000062", 2020-11-29T16:19:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000115", 2020-11-29T16:19:01.335 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000033", 2020-11-29T16:19:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000025", 2020-11-29T16:19:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000123", 2020-11-29T16:19:01.336 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000108", 2020-11-29T16:19:01.337 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-10:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.10:8", 2020-11-29T16:19:01.337 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000094", 2020-11-29T16:19:01.337 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000060", 2020-11-29T16:19:01.338 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000109", 2020-11-29T16:19:01.338 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2.2VAFT+dn/7G.0", 2020-11-29T16:19:01.338 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000013", 2020-11-29T16:19:01.338 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-26:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.26:8", 2020-11-29T16:19:01.339 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-38:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.38:10", 2020-11-29T16:19:01.339 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000051", 2020-11-29T16:19:01.339 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-37:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.37:10", 2020-11-29T16:19:01.339 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000106", 2020-11-29T16:19:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000017", 2020-11-29T16:19:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.f053a404-d473-420f-9a01-38eedc856256.1", 2020-11-29T16:19:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3.+p9m4QZlvxe.0", 2020-11-29T16:19:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000023", 2020-11-29T16:19:01.340 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-18:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.18:1", 2020-11-29T16:19:01.341 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000107", 2020-11-29T16:19:01.341 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000046", 2020-11-29T16:19:01.341 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.2", 2020-11-29T16:19:01.341 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2", 2020-11-29T16:19:01.342 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000067", 2020-11-29T16:19:01.342 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.3", 2020-11-29T16:19:01.342 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000086", 2020-11-29T16:19:01.342 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:5", 2020-11-29T16:19:01.342 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000050", 2020-11-29T16:19:01.343 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0.DCLEOL7wAs1.0", 2020-11-29T16:19:01.343 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000102", 2020-11-29T16:19:01.343 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.e24cddf9-396c-4261-a7bc-b4e1e678325e.1", 2020-11-29T16:19:01.343 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-16:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.16:5", 2020-11-29T16:19:01.344 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000124", 2020-11-29T16:19:01.344 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000001", 2020-11-29T16:19:01.344 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000126", 2020-11-29T16:19:01.345 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000055", 2020-11-29T16:19:01.345 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000061", 2020-11-29T16:19:01.345 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-11:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.11:8", 2020-11-29T16:19:01.345 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000097", 2020-11-29T16:19:01.346 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-19:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.4343.19:8", 2020-11-29T16:19:01.346 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.a1734b2a-fbe5-446c-a86c-68fcf5b64a76.0", 2020-11-29T16:19:01.346 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000042", 2020-11-29T16:19:01.346 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000075", 2020-11-29T16:19:01.347 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000120", 2020-11-29T16:19:01.347 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000030" 2020-11-29T16:19:01.347 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:19:01.347 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-11-29T16:19:01.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:01.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:01.411 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:01.449 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:01.450 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:01.729 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:19:01.729 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:01.729 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-29T16:19:01.729 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.724145Z" 2020-11-29T16:19:01.730 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:01.730 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:01.730 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-29T16:19:01.730 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.727450Z" 2020-11-29T16:19:01.731 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:01.731 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:01.731 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-29T16:19:01.731 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.732810Z" 2020-11-29T16:19:01.731 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:01.732 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:01.733 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-29T16:19:01.733 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.735880Z" 2020-11-29T16:19:01.733 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:01.733 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:19:01.734 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:19:01.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:01.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:01.736 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:01.779 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:01.779 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:02.062 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:19:02.063 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:19:02.063 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:19:02.063 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:19:02.063 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:19:02.064 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 6162550666273865027 2020-11-29T16:19:02.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:02.064 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:19:02.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:02.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:19:02.065 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:02.065 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:02.065 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:19:02.065 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:02.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:02.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:02.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:19:02.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:02.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:02.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:02.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:19:02.067 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:02.067 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:02.067 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006096", 2020-11-29T16:19:02.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:02.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:02.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:02.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:14:38.397271Z" 2020-11-29T16:19:02.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:02.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:02.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:02.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:19:02.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:02.070 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:02.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:19:02.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:02.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:02.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:02.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:19:02.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:02.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:02.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:02.072 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:19:02.072 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:02.072 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:02.072 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:19:02.072 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:02.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:02.073 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:02.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:19:02.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:02.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:02.074 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:19:02.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:02.074 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:19:02.074 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:19:02.074 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:19:02.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:02.075 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:19:02.076 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000006078 target=00000000000000000000:00000000000000003702, shard=1 master=00000000000000000000:00000000000000008472 target=00000000000000000000:00000000000000006096, shard=2 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=3 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374 2020-11-29T16:19:07.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:07.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:07.078 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:07.122 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:07.122 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:07.416 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:19:07.416 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:19:07.417 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:19:07.417 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:19:07.417 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:19:07.417 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 6162550666273865027 2020-11-29T16:19:07.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:07.418 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:19:07.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:07.418 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:19:07.418 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:07.419 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:07.419 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003702", 2020-11-29T16:19:07.419 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:07.419 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:07.419 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:07.420 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:11:23.035547Z" 2020-11-29T16:19:07.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:07.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:07.420 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:07.420 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:19:07.421 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:07.421 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:07.421 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006096", 2020-11-29T16:19:07.421 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:07.422 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:07.422 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:07.422 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:14:38.397271Z" 2020-11-29T16:19:07.422 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:07.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:07.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:07.423 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:19:07.423 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:07.423 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:07.423 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-11-29T16:19:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:08:47.478200Z" 2020-11-29T16:19:07.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:07.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:07.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:07.425 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:19:07.425 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:07.425 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:07.426 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-11-29T16:19:07.426 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:07.426 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:07.426 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:07.426 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:03:56.954413Z" 2020-11-29T16:19:07.427 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:07.427 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:07.427 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:19:07.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:07.427 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:19:07.428 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:19:07.428 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:19:07.428 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:07.428 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:19:07.429 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000006078 target=00000000000000000000:00000000000000003702, shard=1 master=00000000000000000000:00000000000000008472 target=00000000000000000000:00000000000000006096, shard=2 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=3 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374 2020-11-29T16:19:12.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:12.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:12.431 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:12.479 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:12.480 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:12.723 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:19:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:19:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:19:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:19:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:19:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 6162550666273865027 2020-11-29T16:19:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:19:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:19:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-29T16:19:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:18:59.724145Z" 2020-11-29T16:19:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:19:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-29T16:19:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:18:59.727450Z" 2020-11-29T16:19:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:19:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-29T16:19:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:18:59.732810Z" 2020-11-29T16:19:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:19:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-29T16:19:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:18:59.735880Z" 2020-11-29T16:19:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:19:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:19:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:19:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:19:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:12.735 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:19:12.736 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:19:12.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:12.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:12.741 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:12.781 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:12.781 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:13.052 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:19:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-29T16:19:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.724145Z" 2020-11-29T16:19:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.053 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.054 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-29T16:19:13.054 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.727450Z" 2020-11-29T16:19:13.054 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.054 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-29T16:19:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.732810Z" 2020-11-29T16:19:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.055 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.056 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-29T16:19:13.056 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.735880Z" 2020-11-29T16:19:13.056 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:13.056 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:19:13.057 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:19:13.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:13.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:13.059 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:13.139 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:13.140 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:13.398 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:19:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:19:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:19:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:19:13.399 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:19:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 5268709521422156746 2020-11-29T16:19:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:19:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.400 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:19:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-29T16:19:13.401 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:18:59.724145Z" 2020-11-29T16:19:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:13.402 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:19:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:13.403 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-29T16:19:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:13.404 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:18:59.727450Z" 2020-11-29T16:19:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:19:13.405 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-29T16:19:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:13.406 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:18:59.732810Z" 2020-11-29T16:19:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.407 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:19:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-29T16:19:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:13.408 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:18:59.735880Z" 2020-11-29T16:19:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:13.409 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:19:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:19:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:19:13.410 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:19:13.411 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:13.411 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:19:13.412 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:19:13.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:13.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:13.413 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:13.496 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:13.497 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:13.767 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:19:13.767 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.767 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-29T16:19:13.768 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.724145Z" 2020-11-29T16:19:13.768 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.768 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.768 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-29T16:19:13.768 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.727450Z" 2020-11-29T16:19:13.769 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.769 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.769 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-29T16:19:13.770 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.732810Z" 2020-11-29T16:19:13.770 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:13.770 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:13.770 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-29T16:19:13.770 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:18:59.735880Z" 2020-11-29T16:19:13.771 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:13.771 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:19:13.772 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:19:13.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:13.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:13.773 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:13.815 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:13.815 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:14.190 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:19:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:19:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:19:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:19:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:19:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 16089441476439148043 2020-11-29T16:19:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:19:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:19:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:14.192 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006078", 2020-11-29T16:19:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:18:59.724145Z" 2020-11-29T16:19:14.193 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:19:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:14.194 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008472", 2020-11-29T16:19:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:14.195 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:18:59.727450Z" 2020-11-29T16:19:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:14.196 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:19:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-11-29T16:19:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:14.197 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:18:59.732810Z" 2020-11-29T16:19:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:14.198 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:19:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-11-29T16:19:14.199 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:18:59.735880Z" 2020-11-29T16:19:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:14.200 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:19:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:19:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:19:14.201 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:19:14.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:14.202 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:19:14.204 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:19:14.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:14.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:14.205 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:14.284 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:14.285 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:14.557 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:19:14.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-29T16:19:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.008235Z" 2020-11-29T16:19:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:14.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:19:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.015365Z" 2020-11-29T16:19:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:14.559 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-29T16:19:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.021026Z" 2020-11-29T16:19:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:14.560 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003433", 2020-11-29T16:19:14.561 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.002737Z" 2020-11-29T16:19:14.561 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:14.561 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:19:14.562 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:19:14.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:14.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:14.563 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:14.606 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:14.606 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:14.874 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:19:14.874 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:19:14.874 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:19:14.875 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:19:14.875 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:19:14.875 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 201246450793848048 2020-11-29T16:19:14.875 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:14.875 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:19:14.876 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:14.876 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:19:14.876 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:14.876 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:14.877 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-29T16:19:14.877 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:14.877 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:14.877 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:14.877 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.008235Z" 2020-11-29T16:19:14.878 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:14.878 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:14.878 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:14.878 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:19:14.879 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:14.879 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:14.879 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:19:14.879 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:14.880 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:14.880 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:14.880 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.015365Z" 2020-11-29T16:19:14.880 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:14.880 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:14.881 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:14.881 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:19:14.881 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:14.881 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:14.882 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-29T16:19:14.882 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:14.882 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:14.882 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:14.883 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.021026Z" 2020-11-29T16:19:14.883 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:14.883 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:14.883 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:14.884 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:19:14.884 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:14.884 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:14.884 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003433", 2020-11-29T16:19:14.884 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:14.885 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:14.885 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:14.885 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.002737Z" 2020-11-29T16:19:14.885 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:14.886 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:14.886 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:19:14.886 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:14.886 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:19:14.887 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:19:14.887 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:19:14.887 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:14.887 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:19:14.888 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:19:14.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:14.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:14.890 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:14.930 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:14.930 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:15.216 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:19:15.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:15.216 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-29T16:19:15.217 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.008235Z" 2020-11-29T16:19:15.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:15.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:15.217 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:19:15.218 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.015365Z" 2020-11-29T16:19:15.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:15.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:15.218 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-29T16:19:15.219 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.021026Z" 2020-11-29T16:19:15.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:15.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:15.219 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003433", 2020-11-29T16:19:15.220 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.002737Z" 2020-11-29T16:19:15.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:15.220 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:19:15.221 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:19:15.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:15.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:15.223 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:15.266 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:15.267 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:15.529 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:19:15.529 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:19:15.529 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:19:15.530 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:19:15.530 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:19:15.530 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 16876182154419717581 2020-11-29T16:19:15.530 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:15.530 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:19:15.531 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:15.531 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:19:15.531 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:15.531 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:15.532 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-29T16:19:15.532 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:15.532 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:15.532 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:15.532 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.008235Z" 2020-11-29T16:19:15.533 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:15.533 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:15.533 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:15.533 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:19:15.533 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:15.534 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:15.534 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:19:15.534 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:15.534 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:15.534 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:15.535 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.015365Z" 2020-11-29T16:19:15.535 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:15.535 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:15.535 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:15.536 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:19:15.536 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:15.536 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:15.536 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-29T16:19:15.536 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:15.537 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:15.537 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:15.537 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.021026Z" 2020-11-29T16:19:15.537 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:15.537 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:15.538 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:15.538 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:19:15.538 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:15.538 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:15.538 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003433", 2020-11-29T16:19:15.539 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:15.539 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:15.539 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:15.539 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.002737Z" 2020-11-29T16:19:15.539 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:15.540 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:15.540 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:19:15.540 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:15.540 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:19:15.541 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:19:15.541 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:19:15.541 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:15.541 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:19:15.542 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:19:15.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:15.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:15.544 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:15.622 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:15.623 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:15.909 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:19:15.909 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:15.909 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-29T16:19:15.909 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:03.015858Z" 2020-11-29T16:19:15.910 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:15.910 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:15.910 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:19:15.910 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:03.017756Z" 2020-11-29T16:19:15.911 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:15.911 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:15.911 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-29T16:19:15.911 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:03.019906Z" 2020-11-29T16:19:15.911 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:15.912 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:15.912 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-29T16:19:15.912 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:03.013566Z" 2020-11-29T16:19:15.912 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:15.913 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:19:15.913 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:19:15.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:15.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:15.915 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:15.997 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:15.997 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:16.265 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:19:16.265 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:19:16.266 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:19:16.266 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:19:16.266 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:19:16.266 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 3114293326551352523 2020-11-29T16:19:16.267 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:16.267 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:19:16.267 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:16.267 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:19:16.268 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:16.268 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:16.268 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-29T16:19:16.268 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:16.269 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:16.269 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:16.270 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:03.015858Z" 2020-11-29T16:19:16.270 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:16.270 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:16.270 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:16.271 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:19:16.271 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:16.271 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:16.271 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:19:16.271 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:16.272 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:16.272 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:16.272 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:03.017756Z" 2020-11-29T16:19:16.272 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:16.272 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:16.273 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:16.273 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:19:16.273 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:16.273 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:16.274 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-29T16:19:16.274 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:16.274 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:16.274 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:16.275 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:03.019906Z" 2020-11-29T16:19:16.275 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:16.275 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:16.275 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:16.275 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:19:16.275 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:16.276 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:19:16.276 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-29T16:19:16.276 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:16.276 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:16.276 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:16.277 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:03.013566Z" 2020-11-29T16:19:16.277 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:16.277 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:16.277 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:19:16.278 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:16.278 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:19:16.278 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:19:16.278 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:19:16.279 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:16.279 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:19:16.280 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:19:16.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:16.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:16.282 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:16.358 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:16.359 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:16.617 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:19:16.617 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:16.617 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-29T16:19:16.617 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:03.015858Z" 2020-11-29T16:19:16.618 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:16.618 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:16.618 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:19:16.618 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:03.017756Z" 2020-11-29T16:19:16.619 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:16.619 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:16.619 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-29T16:19:16.619 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:03.019906Z" 2020-11-29T16:19:16.620 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:16.620 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:16.620 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-29T16:19:16.620 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:03.013566Z" 2020-11-29T16:19:16.620 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:16.621 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:19:16.622 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:19:16.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:16.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:16.623 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:16.667 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:16.667 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:16.939 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:19:16.940 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:19:16.940 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:19:16.940 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:19:16.941 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:19:16.941 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 2748347558345283993 2020-11-29T16:19:16.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:16.941 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:19:16.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:16.942 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:19:16.942 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:16.942 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:16.942 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005156", 2020-11-29T16:19:16.943 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:16.943 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:16.943 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:16.943 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:03.015858Z" 2020-11-29T16:19:16.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:16.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:16.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:16.944 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:19:16.944 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:16.945 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:16.945 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:19:16.945 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:16.945 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:16.945 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:16.946 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:03.017756Z" 2020-11-29T16:19:16.946 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:16.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:16.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:16.947 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:19:16.947 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:16.947 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:16.947 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001972", 2020-11-29T16:19:16.947 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:16.948 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:16.948 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:16.948 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:03.019906Z" 2020-11-29T16:19:16.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:16.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:16.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:16.949 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:19:16.949 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:16.949 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:19:16.949 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003167", 2020-11-29T16:19:16.950 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:16.950 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:16.950 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:16.950 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:03.013566Z" 2020-11-29T16:19:16.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:19:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:19:16.951 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:19:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:19:16.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:16.952 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:19:16.953 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:19:16.954 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-11-29T16:19:16.954 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-59 2020-11-29T16:19:17.000 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-60 2020-11-29T16:19:17.077 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-61 2020-11-29T16:19:17.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:17.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:17.735 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:17.781 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:17.781 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:18.034 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:19:18.061 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:19:18.062 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:18.062 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:19:18.062 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:19:18.063 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:18.063 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:18.063 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666757.118230_390.1", 2020-11-29T16:19:18.064 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:17.118230Z" 2020-11-29T16:19:18.064 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:18.065 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:18.065 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:19:18.065 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:19:18.065 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:18.066 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:18.066 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:19:18.067 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:19:18.067 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:18.067 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:19:18.069 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:19:18.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:18.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:18.071 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:18.115 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:18.115 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:18.354 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:19:18.355 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:19:18.355 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:19:18.355 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:19:18.356 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:19:18.356 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:19:18.356 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:19:18.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:18.357 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:19:18.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:18.358 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:19:18.358 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:18.358 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:18.359 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:19:18.359 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:18.359 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:19:18.360 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:18.360 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:19:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:19:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:18.363 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:18.363 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666757.109892_388.1", 2020-11-29T16:19:18.363 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:18.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:18.364 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:18.364 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:17.109892Z", 2020-11-29T16:19:18.365 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:19:18.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:18.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:18.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:18.366 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:19:18.366 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:18.367 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:18.367 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:19:18.367 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:18.368 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:18.368 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:18.368 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:19:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:18.370 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:19:18.370 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:18.371 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:18.371 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:19:18.371 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:18.372 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:18.372 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:18.372 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:19:18.373 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:18.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:18.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:18.374 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:19:18.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:18.374 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:19:18.375 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:19:18.375 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:19:18.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:18.376 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:19:18.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666757.118230_390.1 target=1_1606666757.109892_388.1 2020-11-29T16:19:23.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:23.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:23.380 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:23.422 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:23.423 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:23.695 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:19:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:19:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:19:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:19:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:19:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:19:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:19:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:19:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:19:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:19:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:19:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:19:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:19:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666757.118230_390.1", 2020-11-29T16:19:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:17.118230Z", 2020-11-29T16:19:23.702 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:19:23.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:23.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:23.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:23.703 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:19:23.703 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:23.703 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:23.703 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:19:23.703 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:19:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:19:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:19:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:23.706 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:19:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:23.707 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:19:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:19:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:19:23.708 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:19:23.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:23.709 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:19:23.710 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:19:23.710 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:19:23.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:23.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:23.712 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:23.752 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:23.753 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:24.034 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:19:24.035 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:19:24.035 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:19:24.035 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:19:24.035 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:19:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:19:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:19:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:19:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:24.037 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:19:24.037 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:24.037 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:19:24.037 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:19:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:19:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:19:24.038 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:19:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:19:24.039 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:24.040 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:19:24.040 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666757.118230_390.1", 2020-11-29T16:19:24.040 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:24.040 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:24.040 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:24.041 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:17.118230Z", 2020-11-29T16:19:24.041 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:19:24.041 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:24.041 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:24.042 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:24.042 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:19:24.042 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:24.042 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:19:24.042 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:19:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:19:24.043 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:19:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:19:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:19:24.044 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:19:24.045 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:19:24.045 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:19:24.045 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:19:24.045 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:19:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:19:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:19:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:19:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:24.046 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:19:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:19:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:19:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:19:24.047 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:19:24.048 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:19:24.048 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:19:24.049 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:19:24.050 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:19:24.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:24.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:19:24.051 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:19:24.094 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:24.094 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:24.359 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:19:24.359 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:19:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:19:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:19:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:19:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:19:24.360 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:19:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:19:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:19:24.361 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:19:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:24.362 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:19:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:19:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:24.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:19:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:24.364 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666757.118230_390.1", 2020-11-29T16:19:24.365 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:24.365 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:24.365 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:24.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:17.118230Z", 2020-11-29T16:19:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:19:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:24.366 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:19:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:19:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:24.367 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:19:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:24.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:19:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:19:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:19:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:19:24.369 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:19:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:19:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:19:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:19:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:19:24.370 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:19:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:19:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:19:24.371 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:19:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:19:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:19:24.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:19:24.372 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:19:24.373 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:19:24.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:24.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:24.374 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:24.461 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:24.462 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:24.736 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:24.730+0000 7f669d52f080 1 bucket sync caught up with source: 2020-11-29T16:19:24.736 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.51.5, , 00000000001.49.5, , 00000000001.53.5, , , 00000000001.37.5, , , ] 2020-11-29T16:19:24.736 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.51.5, , 00000000001.49.5, , 00000000001.53.5, , , 00000000001.37.5, , , ] 2020-11-29T16:19:24.736 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:24.730+0000 7f669d52f080 0 bucket checkpoint complete 2020-11-29T16:19:24.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:24.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:24.771 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:24.819 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:24.820 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:25.080 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:25.076+0000 7fc0092d6080 1 bucket sync caught up with source: 2020-11-29T16:19:25.080 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.51.5, , 00000000001.49.5, , 00000000001.53.5, , , 00000000001.37.5, , , ] 2020-11-29T16:19:25.081 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.51.5, , 00000000001.49.5, , 00000000001.53.5, , , 00000000001.37.5, , , ] 2020-11-29T16:19:25.081 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:25.076+0000 7fc0092d6080 0 bucket checkpoint complete 2020-11-29T16:19:25.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:25.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:25.111 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:25.158 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:25.158 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:25.404 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:25.398+0000 7fdaad42f080 1 bucket sync caught up with source: 2020-11-29T16:19:25.404 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.51.5, , 00000000001.49.5, , 00000000001.53.5, , , 00000000001.37.5, , , ] 2020-11-29T16:19:25.404 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.51.5, , 00000000001.49.5, , 00000000001.53.5, , , 00000000001.37.5, , , ] 2020-11-29T16:19:25.404 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:25.398+0000 7fdaad42f080 0 bucket checkpoint complete 2020-11-29T16:19:25.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:25.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:25.534 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:25.582 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:25.582 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:25.837 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:25.832+0000 7f2fc01ca080 1 bucket sync caught up with source: 2020-11-29T16:19:25.838 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.51.5, , 00000000001.51.5, , 00000000001.48.5, , , 00000000001.46.5, , , ] 2020-11-29T16:19:25.838 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.51.5, , 00000000001.51.5, , 00000000001.48.5, , , 00000000001.46.5, , , ] 2020-11-29T16:19:25.838 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:25.832+0000 7f2fc01ca080 0 bucket checkpoint complete 2020-11-29T16:19:25.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:25.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:25.871 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:25.915 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:25.916 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:26.165 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:26.160+0000 7f11ab86c080 1 bucket sync caught up with source: 2020-11-29T16:19:26.165 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.51.5, , 00000000001.51.5, , 00000000001.48.5, , , 00000000001.46.5, , , ] 2020-11-29T16:19:26.166 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.51.5, , 00000000001.51.5, , 00000000001.48.5, , , 00000000001.46.5, , , ] 2020-11-29T16:19:26.166 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:26.160+0000 7f11ab86c080 0 bucket checkpoint complete 2020-11-29T16:19:26.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:26.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:26.201 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:26.247 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:26.247 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:26.509 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:26.504+0000 7f02c651c080 1 bucket sync caught up with source: 2020-11-29T16:19:26.509 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.50.5, , 00000000001.50.5, , 00000000001.45.5, , , 00000000001.48.5, , , ] 2020-11-29T16:19:26.510 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.50.5, , 00000000001.50.5, , 00000000001.45.5, , , 00000000001.48.5, , , ] 2020-11-29T16:19:26.510 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:26.504+0000 7f02c651c080 0 bucket checkpoint complete 2020-11-29T16:19:26.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:26.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:26.540 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:26.589 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:26.590 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:26.850 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:26.842+0000 7f5656986080 1 bucket sync caught up with source: 2020-11-29T16:19:26.850 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.50.5, , 00000000001.50.5, , 00000000001.45.5, , , 00000000001.48.5, , , ] 2020-11-29T16:19:26.850 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.50.5, , 00000000001.50.5, , 00000000001.45.5, , , 00000000001.48.5, , , ] 2020-11-29T16:19:26.851 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:26.842+0000 7f5656986080 0 bucket checkpoint complete 2020-11-29T16:19:26.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-59 zones={test-zone2, test-zone1} 2020-11-29T16:19:26.895 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:26.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:26.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:26.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:26.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:26.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:26.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:26.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:26.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:26.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:26.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:27.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:27.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:27.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:27.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:27.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:27.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:27.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:27.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:27.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:27.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:27.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:28.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-59 zones={test-zone2, test-zone1} 2020-11-29T16:19:28.062 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-59 zones={test-zone3, test-zone1} 2020-11-29T16:19:28.072 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:28.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:28.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:28.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:28.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:28.078 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:28.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:28.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:28.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:28.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:28.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:28.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:28.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:28.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:28.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:28.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:28.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:28.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:28.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:28.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:29.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:29.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:29.237 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-59 zones={test-zone3, test-zone1} 2020-11-29T16:19:29.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-59 zones={test-zone3, test-zone2} 2020-11-29T16:19:29.249 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:29.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:29.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:29.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:29.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:29.296 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:29.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:29.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:29.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:29.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:29.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:29.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:29.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:29.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:29.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:29.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:29.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:29.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:30.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:30.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:30.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:30.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:30.449 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-59 zones={test-zone3, test-zone2} 2020-11-29T16:19:30.450 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:30.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:30.451 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:30.495 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:30.495 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:30.769 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:30.762+0000 7fb1730ff080 1 bucket sync caught up with source: 2020-11-29T16:19:30.769 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.48.5, , 00000000001.43.5, , 00000000001.60.5, , , 00000000001.147.5, , , ] 2020-11-29T16:19:30.769 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.48.5, , 00000000001.43.5, , 00000000001.60.5, , , 00000000001.147.5, , , ] 2020-11-29T16:19:30.769 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:30.762+0000 7fb1730ff080 0 bucket checkpoint complete 2020-11-29T16:19:30.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:30.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:30.803 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:30.849 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:30.849 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:31.103 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:31.100+0000 7fa5ac76e080 1 bucket sync caught up with source: 2020-11-29T16:19:31.104 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.48.5, , 00000000001.43.5, , 00000000001.60.5, , , 00000000001.147.5, , , ] 2020-11-29T16:19:31.104 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.48.5, , 00000000001.43.5, , 00000000001.60.5, , , 00000000001.147.5, , , ] 2020-11-29T16:19:31.104 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:31.100+0000 7fa5ac76e080 0 bucket checkpoint complete 2020-11-29T16:19:31.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:31.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:31.131 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:31.180 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:31.434 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:31.426+0000 7f12fea6f080 1 bucket sync caught up with source: 2020-11-29T16:19:31.434 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.48.5, , 00000000001.43.5, , 00000000001.60.5, , , 00000000001.147.5, , , ] 2020-11-29T16:19:31.435 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.48.5, , 00000000001.43.5, , 00000000001.60.5, , , 00000000001.147.5, , , ] 2020-11-29T16:19:31.435 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:31.426+0000 7f12fea6f080 0 bucket checkpoint complete 2020-11-29T16:19:31.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:31.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:31.550 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:31.596 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:31.596 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:31.855 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:31.852+0000 7fc69adac080 1 bucket sync caught up with source: 2020-11-29T16:19:31.856 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.53.5, , 00000000001.50.5, , 00000000001.69.5, , , 00000000001.162.5, , , ] 2020-11-29T16:19:31.856 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.53.5, , 00000000001.50.5, , 00000000001.69.5, , , 00000000001.162.5, , , ] 2020-11-29T16:19:31.856 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:31.852+0000 7fc69adac080 0 bucket checkpoint complete 2020-11-29T16:19:31.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:31.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:31.889 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:31.934 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:31.934 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:32.176 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:32.172+0000 7facff264080 1 bucket sync caught up with source: 2020-11-29T16:19:32.177 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.53.5, , 00000000001.50.5, , 00000000001.69.5, , , 00000000001.162.5, , , ] 2020-11-29T16:19:32.177 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.53.5, , 00000000001.50.5, , 00000000001.69.5, , , 00000000001.162.5, , , ] 2020-11-29T16:19:32.177 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:32.172+0000 7facff264080 0 bucket checkpoint complete 2020-11-29T16:19:32.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:32.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:32.207 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:32.294 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:32.295 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:32.518 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:32.512+0000 7fc5e1e1c080 1 bucket sync caught up with source: 2020-11-29T16:19:32.518 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.56.5, , 00000000001.53.5, , 00000000001.79.5, , , 00000000001.160.5, , , ] 2020-11-29T16:19:32.518 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.56.5, , 00000000001.53.5, , 00000000001.79.5, , , 00000000001.160.5, , , ] 2020-11-29T16:19:32.518 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:32.512+0000 7fc5e1e1c080 0 bucket checkpoint complete 2020-11-29T16:19:32.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:32.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:32.551 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:32.599 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:32.599 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:32.862 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:32.854+0000 7f6f732bf080 1 bucket sync caught up with source: 2020-11-29T16:19:32.863 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.56.5, , 00000000001.53.5, , 00000000001.79.5, , , 00000000001.160.5, , , ] 2020-11-29T16:19:32.863 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.56.5, , 00000000001.53.5, , 00000000001.79.5, , , 00000000001.160.5, , , ] 2020-11-29T16:19:32.863 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:32.854+0000 7f6f732bf080 0 bucket checkpoint complete 2020-11-29T16:19:32.897 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-60 zones={test-zone2, test-zone1} 2020-11-29T16:19:32.908 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:32.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:32.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:32.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:32.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:32.916 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:32.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:32.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:32.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:32.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:32.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:32.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:33.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:33.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:33.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:33.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:33.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:33.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:33.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:33.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:33.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:33.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:34.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-60 zones={test-zone2, test-zone1} 2020-11-29T16:19:34.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-60 zones={test-zone3, test-zone1} 2020-11-29T16:19:34.028 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:34.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:34.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:34.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:34.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:34.035 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:34.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:34.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:34.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:34.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:34.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:34.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:34.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:34.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:34.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:34.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:34.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:34.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:34.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:34.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:34.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:35.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:35.145 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-60 zones={test-zone3, test-zone1} 2020-11-29T16:19:35.145 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-60 zones={test-zone3, test-zone2} 2020-11-29T16:19:35.157 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:35.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:35.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:35.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:35.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:35.164 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:35.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:35.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:35.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:35.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:35.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:35.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:35.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:35.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:35.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:35.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:35.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:35.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:35.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:36.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:36.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:36.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:36.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-60 zones={test-zone3, test-zone2} 2020-11-29T16:19:36.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:36.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:36.304 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:36.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:36.349 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:36.619 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:36.614+0000 7f6e5ca3c080 1 bucket sync caught up with source: 2020-11-29T16:19:36.619 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.45.5, , 00000000001.149.5, , 00000000001.37.5, , , 00000000001.51.5, , , ] 2020-11-29T16:19:36.620 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.45.5, , 00000000001.149.5, , 00000000001.37.5, , , 00000000001.51.5, , , ] 2020-11-29T16:19:36.620 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:36.614+0000 7f6e5ca3c080 0 bucket checkpoint complete 2020-11-29T16:19:36.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:36.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:36.651 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:36.942 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:36.935+0000 7f927205a080 1 bucket sync caught up with source: 2020-11-29T16:19:36.942 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.45.5, , 00000000001.149.5, , 00000000001.37.5, , , 00000000001.51.5, , , ] 2020-11-29T16:19:36.943 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.45.5, , 00000000001.149.5, , 00000000001.37.5, , , 00000000001.51.5, , , ] 2020-11-29T16:19:36.943 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:36.935+0000 7f927205a080 0 bucket checkpoint complete 2020-11-29T16:19:36.971 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:36.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:36.972 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:37.021 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:37.021 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:37.264 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:37.258+0000 7f8d6163c080 1 bucket sync caught up with source: 2020-11-29T16:19:37.264 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.45.5, , 00000000001.149.5, , 00000000001.37.5, , , 00000000001.51.5, , , ] 2020-11-29T16:19:37.265 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.45.5, , 00000000001.149.5, , 00000000001.37.5, , , 00000000001.51.5, , , ] 2020-11-29T16:19:37.265 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:37.258+0000 7f8d6163c080 0 bucket checkpoint complete 2020-11-29T16:19:37.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:37.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:37.391 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:37.434 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:37.435 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:37.674 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:37.667+0000 7f1e8673d080 1 bucket sync caught up with source: 2020-11-29T16:19:37.675 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.52.5, , 00000000001.164.5, , 00000000001.38.5, , , 00000000001.53.5, , , ] 2020-11-29T16:19:37.675 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.52.5, , 00000000001.164.5, , 00000000001.38.5, , , 00000000001.53.5, , , ] 2020-11-29T16:19:37.675 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:37.667+0000 7f1e8673d080 0 bucket checkpoint complete 2020-11-29T16:19:37.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:37.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:37.709 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:37.798 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:37.798 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:38.045 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:38.039+0000 7f0ea173f080 1 bucket sync caught up with source: 2020-11-29T16:19:38.045 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.52.5, , 00000000001.164.5, , 00000000001.38.5, , , 00000000001.53.5, , , ] 2020-11-29T16:19:38.046 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.52.5, , 00000000001.164.5, , 00000000001.38.5, , , 00000000001.53.5, , , ] 2020-11-29T16:19:38.046 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:38.039+0000 7f0ea173f080 0 bucket checkpoint complete 2020-11-29T16:19:38.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:38.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:38.078 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:38.165 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:38.166 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:38.420 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:38.415+0000 7fbd0c437080 1 bucket sync caught up with source: 2020-11-29T16:19:38.421 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.55.5, , 00000000001.158.5, , 00000000001.37.5, , , 00000000001.52.5, , , ] 2020-11-29T16:19:38.421 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.55.5, , 00000000001.158.5, , 00000000001.37.5, , , 00000000001.52.5, , , ] 2020-11-29T16:19:38.421 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:38.415+0000 7fbd0c437080 0 bucket checkpoint complete 2020-11-29T16:19:38.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:38.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:38.449 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:38.496 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:38.762 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:38.754+0000 7fd239f42080 1 bucket sync caught up with source: 2020-11-29T16:19:38.763 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.55.5, , 00000000001.158.5, , 00000000001.37.5, , , 00000000001.52.5, , , ] 2020-11-29T16:19:38.763 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.55.5, , 00000000001.158.5, , 00000000001.37.5, , , 00000000001.52.5, , , ] 2020-11-29T16:19:38.763 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:38.754+0000 7fd239f42080 0 bucket checkpoint complete 2020-11-29T16:19:38.796 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-61 zones={test-zone2, test-zone1} 2020-11-29T16:19:38.806 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:38.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:38.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:38.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:38.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:38.856 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:38.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:38.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:38.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:38.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:38.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:38.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:39.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:39.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:39.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:39.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:39.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:39.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:39.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:39.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:39.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:39.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:39.977 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-61 zones={test-zone2, test-zone1} 2020-11-29T16:19:39.978 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-61 zones={test-zone3, test-zone1} 2020-11-29T16:19:39.987 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:39.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:39.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:39.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:39.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:39.993 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:39.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:39.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:39.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:39.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:40.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:40.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:40.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:40.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:40.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:40.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:40.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:40.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:40.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:40.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:40.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:40.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:41.097 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-61 zones={test-zone3, test-zone1} 2020-11-29T16:19:41.098 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-61 zones={test-zone3, test-zone2} 2020-11-29T16:19:41.109 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:19:41.113 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:41.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:41.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:41.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:41.115 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:19:41.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:19:41.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:19:41.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:19:41.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:19:41.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:41.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:41.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:19:41.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:41.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:41.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:19:41.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:41.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:41.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:19:41.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:42.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:42.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:19:42.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-61 zones={test-zone3, test-zone2} 2020-11-29T16:19:42.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:42.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:42.243 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:19:42.285 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:42.286 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:42.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kcmyep-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:42.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kcmyep-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:42.628 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kcmyep-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:19:42.713 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:42.713 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:43.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:43.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:43.085 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:19:43.167 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:43.167 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:43.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kcmyep-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:43.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kcmyep-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:43.485 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kcmyep-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:19:43.573 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:43.574 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:43.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:43.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:43.895 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:19:43.934 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:43.935 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:44.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kcmyep-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:44.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kcmyep-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:19:44.272 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kcmyep-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:19:44.356 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:44.356 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:45.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:45.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:45.251 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:45.299 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:45.299 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:45.552 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:45.546+0000 7f451e560080 1 waiting for incremental sync to catch up: 2020-11-29T16:19:45.552 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.57.5, , 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, , 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, , 00000000004.39.1] 2020-11-29T16:19:45.553 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:19:50.557 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:50.550+0000 7f451e560080 1 waiting for incremental sync to catch up: 2020-11-29T16:19:50.558 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.57.5, , 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, , 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, , 00000000004.39.1] 2020-11-29T16:19:50.558 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:19:55.562 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:55.554+0000 7f451e560080 1 bucket sync caught up with source: 2020-11-29T16:19:55.562 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:19:55.563 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:19:55.563 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:19:55.554+0000 7f451e560080 0 bucket checkpoint complete 2020-11-29T16:19:55.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:55.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:19:55.593 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:19:55.641 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:19:55.641 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:19:55.889 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:19:55.883+0000 7f6fad4e7080 1 waiting for incremental sync to catch up: 2020-11-29T16:19:55.890 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.57.5, , 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, , 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, , 00000000004.39.1] 2020-11-29T16:19:55.890 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:20:00.894 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:00.887+0000 7f6fad4e7080 1 waiting for incremental sync to catch up: 2020-11-29T16:20:00.895 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.57.5, , 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, , 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, , 00000000004.39.1] 2020-11-29T16:20:00.895 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:20:05.899 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:05.895+0000 7f6fad4e7080 1 bucket sync caught up with source: 2020-11-29T16:20:05.900 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:20:05.900 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:20:05.900 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:05.895+0000 7f6fad4e7080 0 bucket checkpoint complete 2020-11-29T16:20:05.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:05.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:05.934 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:05.983 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:05.983 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:06.249 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:06.241+0000 7fd48a83b080 1 bucket sync caught up with source: 2020-11-29T16:20:06.249 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:20:06.249 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.57.5, 00000000004.48.1, 00000000006.61.5, 00000000004.44.1, 00000000006.59.5, 00000000004.52.1, 00000000004.38.1, 00000000006.51.5, 00000000004.111.1, 00000000004.47.1, 00000000004.39.1] 2020-11-29T16:20:06.250 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:06.241+0000 7fd48a83b080 0 bucket checkpoint complete 2020-11-29T16:20:06.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:06.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:06.363 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:06.410 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:06.410 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:06.681 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:06.675+0000 7fd124a29080 1 bucket sync caught up with source: 2020-11-29T16:20:06.681 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000002.53.5, , 00000000002.55.5, , 00000000002.50.5, , , 00000000002.48.5, , , ] 2020-11-29T16:20:06.681 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000002.53.5, , 00000000002.55.5, , 00000000002.50.5, , , 00000000002.48.5, , , ] 2020-11-29T16:20:06.682 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:06.675+0000 7fd124a29080 0 bucket checkpoint complete 2020-11-29T16:20:06.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:06.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:06.713 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:06.758 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:06.759 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:07.020 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:07.015+0000 7f19646ee080 1 bucket sync caught up with source: 2020-11-29T16:20:07.021 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000002.53.5, , 00000000002.55.5, , 00000000002.50.5, , , 00000000002.48.5, , , ] 2020-11-29T16:20:07.021 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000002.53.5, , 00000000002.55.5, , 00000000002.50.5, , , 00000000002.48.5, , , ] 2020-11-29T16:20:07.021 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:07.015+0000 7f19646ee080 0 bucket checkpoint complete 2020-11-29T16:20:07.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:07.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:07.053 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:07.143 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:07.143 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:07.393 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:07.387+0000 7f958ed69080 1 bucket sync caught up with source: 2020-11-29T16:20:07.393 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.56.5, 00000000002.47.1, 00000000004.60.5, 00000000002.51.1, 00000000004.51.5, 00000000002.60.1, 00000000002.36.1, 00000000004.60.5, 00000000002.136.1, 00000000002.45.1, 00000000002.42.1] 2020-11-29T16:20:07.393 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.56.5, 00000000002.47.1, 00000000004.60.5, 00000000002.51.1, 00000000004.51.5, 00000000002.60.1, 00000000002.36.1, 00000000004.60.5, 00000000002.136.1, 00000000002.45.1, 00000000002.42.1] 2020-11-29T16:20:07.394 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:07.387+0000 7f958ed69080 0 bucket checkpoint complete 2020-11-29T16:20:07.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:07.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-59', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:07.423 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-59 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:07.473 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:07.474 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:07.733 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:07.725+0000 7fa77f253080 1 bucket sync caught up with source: 2020-11-29T16:20:07.734 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000004.56.5, 00000000002.47.1, 00000000004.60.5, 00000000002.51.1, 00000000004.51.5, 00000000002.60.1, 00000000002.36.1, 00000000004.60.5, 00000000002.136.1, 00000000002.45.1, 00000000002.42.1] 2020-11-29T16:20:07.734 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000004.56.5, 00000000002.47.1, 00000000004.60.5, 00000000002.51.1, 00000000004.51.5, 00000000002.60.1, 00000000002.36.1, 00000000004.60.5, 00000000002.136.1, 00000000002.45.1, 00000000002.42.1] 2020-11-29T16:20:07.734 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:07.725+0000 7fa77f253080 0 bucket checkpoint complete 2020-11-29T16:20:07.768 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-59 zones={test-zone2, test-zone1} 2020-11-29T16:20:07.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:07.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:07.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:07.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:07.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:07.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:07.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:07.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:07.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:07.794 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:07.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:07.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:07.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:07.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:07.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:07.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:07.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:07.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:07.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:07.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:08.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:08.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:08.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:08.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:08.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:08.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:08.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:08.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:08.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:08.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:08.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:09.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:09.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:09.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:09.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:09.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:09.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:09.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:09.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:09.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:09.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:09.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:10.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-59 zones={test-zone2, test-zone1} 2020-11-29T16:20:10.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-59 zones={test-zone3, test-zone1} 2020-11-29T16:20:10.087 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:10.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:10.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:10.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:10.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:10.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:10.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:10.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:10.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:10.098 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:10.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:10.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:10.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:10.107 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:10.108 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:10.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:10.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:10.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:10.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:10.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:10.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:10.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:10.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:10.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:10.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:10.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:10.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:10.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:11.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:11.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:11.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:11.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:11.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:11.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:11.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:11.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:11.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:11.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:11.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:12.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:12.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:12.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:12.351 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-59 zones={test-zone3, test-zone1} 2020-11-29T16:20:12.352 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-59 zones={test-zone3, test-zone2} 2020-11-29T16:20:12.362 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:12.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:12.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:12.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:12.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:12.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:12.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:12.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:12.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:12.369 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:12.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:12.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:12.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:12.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:12.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:12.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:12.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:12.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:12.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:12.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:12.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:12.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:12.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:12.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:12.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:12.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:13.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:13.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:13.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:13.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:13.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:13.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:13.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:13.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:13.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:13.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:14.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:14.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:14.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:14.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:14.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:14.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:14.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-59 zones={test-zone3, test-zone2} 2020-11-29T16:20:14.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:14.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:14.587 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:14.634 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:14.634 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:14.894 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:14.885+0000 7f569f2a4080 1 bucket sync caught up with source: 2020-11-29T16:20:14.894 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.58.5, 00000000004.54.1, 00000000006.55.5, 00000000004.67.1, 00000000006.74.5, 00000000004.52.1, 00000000004.58.1, 00000000006.159.5, 00000000004.41.1, 00000000004.55.1, 00000000004.42.1] 2020-11-29T16:20:14.894 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.58.5, 00000000004.54.1, 00000000006.55.5, 00000000004.67.1, 00000000006.74.5, 00000000004.52.1, 00000000004.58.1, 00000000006.159.5, 00000000004.41.1, 00000000004.55.1, 00000000004.42.1] 2020-11-29T16:20:14.894 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:14.885+0000 7f569f2a4080 0 bucket checkpoint complete 2020-11-29T16:20:14.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:14.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:14.932 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:14.980 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:14.981 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:15.247 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:15.243+0000 7f7f5f36a080 1 bucket sync caught up with source: 2020-11-29T16:20:15.247 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.58.5, 00000000004.54.1, 00000000006.55.5, 00000000004.67.1, 00000000006.74.5, 00000000004.52.1, 00000000004.58.1, 00000000006.159.5, 00000000004.41.1, 00000000004.55.1, 00000000004.42.1] 2020-11-29T16:20:15.248 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.58.5, 00000000004.54.1, 00000000006.55.5, 00000000004.67.1, 00000000006.74.5, 00000000004.52.1, 00000000004.58.1, 00000000006.159.5, 00000000004.41.1, 00000000004.55.1, 00000000004.42.1] 2020-11-29T16:20:15.248 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:15.243+0000 7f7f5f36a080 0 bucket checkpoint complete 2020-11-29T16:20:15.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:15.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:15.276 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:15.324 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:15.324 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:15.565 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:15.557+0000 7f4eeb864080 1 bucket sync caught up with source: 2020-11-29T16:20:15.566 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.58.5, 00000000004.54.1, 00000000006.55.5, 00000000004.67.1, 00000000006.74.5, 00000000004.52.1, 00000000004.58.1, 00000000006.159.5, 00000000004.41.1, 00000000004.55.1, 00000000004.42.1] 2020-11-29T16:20:15.566 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.58.5, 00000000004.54.1, 00000000006.55.5, 00000000004.67.1, 00000000006.74.5, 00000000004.52.1, 00000000004.58.1, 00000000006.159.5, 00000000004.41.1, 00000000004.55.1, 00000000004.42.1] 2020-11-29T16:20:15.567 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:15.557+0000 7f4eeb864080 0 bucket checkpoint complete 2020-11-29T16:20:15.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:15.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:15.700 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:15.747 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:15.748 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:16.028 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:16.023+0000 7fe53234c080 1 bucket sync caught up with source: 2020-11-29T16:20:16.029 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000002.55.5, , 00000000002.54.5, , 00000000002.71.5, , , 00000000002.166.5, , , ] 2020-11-29T16:20:16.029 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000002.55.5, , 00000000002.54.5, , 00000000002.71.5, , , 00000000002.166.5, , , ] 2020-11-29T16:20:16.030 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:16.023+0000 7fe53234c080 0 bucket checkpoint complete 2020-11-29T16:20:16.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:16.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:16.061 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:16.109 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:16.109 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:16.376 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:16.371+0000 7f8f16415080 1 bucket sync caught up with source: 2020-11-29T16:20:16.377 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000002.55.5, , 00000000002.54.5, , 00000000002.71.5, , , 00000000002.166.5, , , ] 2020-11-29T16:20:16.377 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000002.55.5, , 00000000002.54.5, , 00000000002.71.5, , , 00000000002.166.5, , , ] 2020-11-29T16:20:16.377 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:16.371+0000 7f8f16415080 0 bucket checkpoint complete 2020-11-29T16:20:16.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:16.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:16.409 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:16.453 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:16.453 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:16.718 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:16.711+0000 7fd85042f080 1 bucket sync caught up with source: 2020-11-29T16:20:16.719 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.66.5, 00000000002.48.1, 00000000004.67.5, 00000000002.82.1, 00000000004.93.5, 00000000002.52.1, 00000000002.70.1, 00000000004.172.5, 00000000002.40.1, 00000000002.50.1, 00000000002.38.1] 2020-11-29T16:20:16.719 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.66.5, 00000000002.48.1, 00000000004.67.5, 00000000002.82.1, 00000000004.93.5, 00000000002.52.1, 00000000002.70.1, 00000000004.172.5, 00000000002.40.1, 00000000002.50.1, 00000000002.38.1] 2020-11-29T16:20:16.720 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:16.711+0000 7fd85042f080 0 bucket checkpoint complete 2020-11-29T16:20:16.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:16.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-60', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:16.748 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-60 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:16.795 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:16.796 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:17.046 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:17.037+0000 7f70c4763080 1 bucket sync caught up with source: 2020-11-29T16:20:17.047 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000004.66.5, 00000000002.48.1, 00000000004.67.5, 00000000002.82.1, 00000000004.93.5, 00000000002.52.1, 00000000002.70.1, 00000000004.172.5, 00000000002.40.1, 00000000002.50.1, 00000000002.38.1] 2020-11-29T16:20:17.047 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000004.66.5, 00000000002.48.1, 00000000004.67.5, 00000000002.82.1, 00000000004.93.5, 00000000002.52.1, 00000000002.70.1, 00000000004.172.5, 00000000002.40.1, 00000000002.50.1, 00000000002.38.1] 2020-11-29T16:20:17.048 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:17.037+0000 7f70c4763080 0 bucket checkpoint complete 2020-11-29T16:20:17.080 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-60 zones={test-zone2, test-zone1} 2020-11-29T16:20:17.093 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:17.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:17.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:17.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:17.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:17.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:17.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:17.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:17.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:17.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:17.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:17.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:17.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:17.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:17.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:17.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:17.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:17.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:17.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:17.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:17.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:17.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:17.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:17.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:17.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:17.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:17.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:17.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:18.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:18.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:18.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:18.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:18.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:18.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:18.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:18.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:18.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:18.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:18.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:19.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:19.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:19.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:19.370 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-60 zones={test-zone2, test-zone1} 2020-11-29T16:20:19.371 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-60 zones={test-zone3, test-zone1} 2020-11-29T16:20:19.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:19.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:19.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:19.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:19.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:19.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:19.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:19.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:19.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:19.391 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:19.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:19.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:19.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:19.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:19.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:19.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:19.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:19.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:19.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:19.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:19.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:19.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:19.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:19.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:19.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:20.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:20.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:20.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:20.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:20.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:20.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:20.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:20.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:20.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:20.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:20.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:21.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:21.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:21.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:21.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:21.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:21.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:21.637 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-60 zones={test-zone3, test-zone1} 2020-11-29T16:20:21.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-60 zones={test-zone3, test-zone2} 2020-11-29T16:20:21.649 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:21.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:21.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:21.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:21.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:21.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:21.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:21.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:21.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:21.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:21.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:21.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:21.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:21.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:21.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:21.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:21.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:21.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:21.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:21.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:21.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:21.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:22.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:22.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:22.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:22.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:22.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:22.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:22.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:22.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:22.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:22.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:22.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:23.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:23.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:23.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:23.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:23.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:23.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:23.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:23.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:23.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:23.914 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-60 zones={test-zone3, test-zone2} 2020-11-29T16:20:23.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:23.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:23.916 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:23.957 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:23.957 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:24.213 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:24.205+0000 7ff46ed85080 1 bucket sync caught up with source: 2020-11-29T16:20:24.213 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.57.5, 00000000004.44.1, 00000000006.161.5, 00000000004.45.1, 00000000006.47.5, 00000000004.32.1, 00000000004.43.1, 00000000007.65.5, 00000000004.49.1, 00000000004.72.1, 00000000004.115.1] 2020-11-29T16:20:24.213 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.57.5, 00000000004.44.1, 00000000006.161.5, 00000000004.45.1, 00000000006.47.5, 00000000004.32.1, 00000000004.43.1, 00000000007.65.5, 00000000004.49.1, 00000000004.72.1, 00000000004.115.1] 2020-11-29T16:20:24.214 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:24.205+0000 7ff46ed85080 0 bucket checkpoint complete 2020-11-29T16:20:24.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:24.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:24.246 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:24.291 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:24.292 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:24.535 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:24.531+0000 7fc2a7a0c080 1 bucket sync caught up with source: 2020-11-29T16:20:24.536 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.57.5, 00000000004.44.1, 00000000006.161.5, 00000000004.45.1, 00000000006.47.5, 00000000004.32.1, 00000000004.43.1, 00000000007.65.5, 00000000004.49.1, 00000000004.72.1, 00000000004.115.1] 2020-11-29T16:20:24.536 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.57.5, 00000000004.44.1, 00000000006.161.5, 00000000004.45.1, 00000000006.47.5, 00000000004.32.1, 00000000004.43.1, 00000000007.65.5, 00000000004.49.1, 00000000004.72.1, 00000000004.115.1] 2020-11-29T16:20:24.536 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:24.531+0000 7fc2a7a0c080 0 bucket checkpoint complete 2020-11-29T16:20:24.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:24.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:24.564 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:24.611 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:24.611 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:24.856 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:24.849+0000 7f93f9771080 1 bucket sync caught up with source: 2020-11-29T16:20:24.857 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.57.5, 00000000004.44.1, 00000000006.161.5, 00000000004.45.1, 00000000006.47.5, 00000000004.32.1, 00000000004.43.1, 00000000007.65.5, 00000000004.49.1, 00000000004.72.1, 00000000004.115.1] 2020-11-29T16:20:24.857 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.57.5, 00000000004.44.1, 00000000006.161.5, 00000000004.45.1, 00000000006.47.5, 00000000004.32.1, 00000000004.43.1, 00000000007.65.5, 00000000004.49.1, 00000000004.72.1, 00000000004.115.1] 2020-11-29T16:20:24.857 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:24.849+0000 7f93f9771080 0 bucket checkpoint complete 2020-11-29T16:20:24.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:24.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:24.981 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:25.030 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:25.030 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:25.289 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:25.283+0000 7f684c9d4080 1 bucket sync caught up with source: 2020-11-29T16:20:25.289 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000002.59.5, , 00000000002.171.5, , 00000000002.42.5, , , 00000000002.60.5, , , ] 2020-11-29T16:20:25.290 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000002.59.5, , 00000000002.171.5, , 00000000002.42.5, , , 00000000002.60.5, , , ] 2020-11-29T16:20:25.290 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:25.283+0000 7f684c9d4080 0 bucket checkpoint complete 2020-11-29T16:20:25.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:25.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:25.325 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:25.376 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:25.376 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:25.614 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:25.607+0000 7f548311d080 1 bucket sync caught up with source: 2020-11-29T16:20:25.614 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000002.59.5, , 00000000002.171.5, , 00000000002.42.5, , , 00000000002.60.5, , , ] 2020-11-29T16:20:25.615 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000002.59.5, , 00000000002.171.5, , 00000000002.42.5, , , 00000000002.60.5, , , ] 2020-11-29T16:20:25.615 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:25.607+0000 7f548311d080 0 bucket checkpoint complete 2020-11-29T16:20:25.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:25.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:25.649 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:25.697 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:25.697 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:25.957 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:25.951+0000 7fa88b614080 1 bucket sync caught up with source: 2020-11-29T16:20:25.958 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.64.1, 00000000002.44.1, 00000000004.169.1, 00000000002.44.1, 00000000004.45.1, 00000000002.37.1, 00000000002.52.1, 00000000004.63.1, 00000000002.61.1, 00000000002.89.1, 00000000002.140.1] 2020-11-29T16:20:25.958 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.64.1, 00000000002.44.1, 00000000004.169.1, 00000000002.44.1, 00000000004.45.1, 00000000002.37.1, 00000000002.52.1, 00000000004.63.1, 00000000002.61.1, 00000000002.89.1, 00000000002.140.1] 2020-11-29T16:20:25.958 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:25.951+0000 7fa88b614080 0 bucket checkpoint complete 2020-11-29T16:20:25.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:25.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-61', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:25.987 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-61 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:26.032 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:26.032 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:26.294 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:26.285+0000 7efd2cd57080 1 bucket sync caught up with source: 2020-11-29T16:20:26.295 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000004.64.1, 00000000002.44.1, 00000000004.169.1, 00000000002.44.1, 00000000004.45.1, 00000000002.37.1, 00000000002.52.1, 00000000004.63.1, 00000000002.61.1, 00000000002.89.1, 00000000002.140.1] 2020-11-29T16:20:26.295 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000004.64.1, 00000000002.44.1, 00000000004.169.1, 00000000002.44.1, 00000000004.45.1, 00000000002.37.1, 00000000002.52.1, 00000000004.63.1, 00000000002.61.1, 00000000002.89.1, 00000000002.140.1] 2020-11-29T16:20:26.295 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:26.285+0000 7efd2cd57080 0 bucket checkpoint complete 2020-11-29T16:20:26.325 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-61 zones={test-zone2, test-zone1} 2020-11-29T16:20:26.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:26.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:26.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:26.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:26.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:26.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:26.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:26.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:26.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:26.343 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:26.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:26.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:26.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:26.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:26.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:26.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:26.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:26.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:26.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:26.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:26.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:26.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:26.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:26.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:26.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:26.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:27.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:27.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:27.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:27.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:27.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:27.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:27.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:27.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:27.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:27.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:28.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:28.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:28.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:28.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:28.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:28.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:28.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-61 zones={test-zone2, test-zone1} 2020-11-29T16:20:28.586 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-61 zones={test-zone3, test-zone1} 2020-11-29T16:20:28.596 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:28.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:28.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:28.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:28.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:28.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:28.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:28.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:28.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:28.604 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:28.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:28.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:28.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:28.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:28.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:28.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:28.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:28.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:28.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:28.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:28.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:28.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:28.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:29.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:29.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:29.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:29.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:29.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:29.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:29.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:29.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:29.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:29.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:29.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:30.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:30.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:30.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:30.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:30.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:30.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:30.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:30.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:30.818 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-61 zones={test-zone3, test-zone1} 2020-11-29T16:20:30.819 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-61 zones={test-zone3, test-zone2} 2020-11-29T16:20:30.829 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:30.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:30.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:30.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:30.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:30.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:30.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:30.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:30.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:30.839 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:30.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:30.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:30.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:30.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:30.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:20:30.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:20:30.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:20:30.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:20:30.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:30.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:31.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:31.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:31.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:31.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:31.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:31.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:31.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:31.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:31.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:31.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:32.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:32.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:32.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:20:32.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:32.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:32.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:20:32.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:32.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:32.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:20:32.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:32.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:32.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:20:33.114 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-61 zones={test-zone3, test-zone2} 2020-11-29T16:20:33.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:33.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:33.115 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:33.157 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:33.158 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:33.447 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:20:33.447 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:33.447 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-29T16:20:33.447 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:52.657461Z" 2020-11-29T16:20:33.448 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:33.448 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:33.448 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-29T16:20:33.449 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:44.658895Z" 2020-11-29T16:20:33.449 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:33.449 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:33.449 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-29T16:20:33.449 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:52.638714Z" 2020-11-29T16:20:33.450 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:33.450 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:33.450 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008081", 2020-11-29T16:20:33.450 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:09.181523Z" 2020-11-29T16:20:33.450 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:33.451 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:20:33.452 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:20:33.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:33.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:33.453 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:33.495 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:33.496 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:33.777 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:20:33.778 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:20:33.778 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:20:33.778 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:20:33.779 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:20:33.779 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 6162550666273865027 2020-11-29T16:20:33.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:33.779 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:20:33.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:33.780 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:20:33.780 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:33.780 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:33.780 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-29T16:20:33.781 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:33.781 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:33.781 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:33.781 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:52.657461Z" 2020-11-29T16:20:33.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:33.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:33.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:33.782 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:20:33.782 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:33.782 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:33.783 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-29T16:20:33.783 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:33.783 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:33.783 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:33.784 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:44.658895Z" 2020-11-29T16:20:33.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:33.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:33.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:33.784 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:20:33.785 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:33.785 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:33.785 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-29T16:20:33.785 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:33.785 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:33.786 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:33.786 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:52.638714Z" 2020-11-29T16:20:33.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:33.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:33.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:33.787 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:20:33.787 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:33.787 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:33.787 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008081", 2020-11-29T16:20:33.787 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:33.788 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:33.788 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:33.788 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:09.181523Z" 2020-11-29T16:20:33.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:33.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:33.789 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:20:33.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:33.789 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:20:33.789 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:20:33.790 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:20:33.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:33.790 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:20:33.791 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:20:33.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:33.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:33.793 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:33.832 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:33.832 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:34.111 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:20:34.112 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.112 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-29T16:20:34.112 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:52.657461Z" 2020-11-29T16:20:34.112 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.113 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.113 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-29T16:20:34.113 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:44.658895Z" 2020-11-29T16:20:34.113 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.113 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.114 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-29T16:20:34.114 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:52.638714Z" 2020-11-29T16:20:34.114 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.114 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.115 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008081", 2020-11-29T16:20:34.115 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:09.181523Z" 2020-11-29T16:20:34.115 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:34.115 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:20:34.116 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:20:34.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:34.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:34.118 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:34.199 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:34.200 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:34.480 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:20:34.481 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:20:34.481 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:20:34.481 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:20:34.482 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:20:34.482 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 5268709521422156746 2020-11-29T16:20:34.482 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.482 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:20:34.482 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.483 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:20:34.483 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:34.483 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:34.483 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-29T16:20:34.484 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:34.484 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:34.484 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:34.484 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:52.657461Z" 2020-11-29T16:20:34.484 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:34.485 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.485 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.485 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:20:34.485 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:34.485 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:34.486 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-29T16:20:34.486 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:34.486 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:34.487 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:34.487 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:44.658895Z" 2020-11-29T16:20:34.487 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:34.487 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.487 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.488 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:20:34.488 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:34.488 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:34.488 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-29T16:20:34.488 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:34.489 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:34.489 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:34.489 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:52.638714Z" 2020-11-29T16:20:34.489 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:34.490 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.490 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.490 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:20:34.490 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:34.490 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:34.491 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008081", 2020-11-29T16:20:34.491 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:34.491 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:34.491 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:34.491 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:20:09.181523Z" 2020-11-29T16:20:34.492 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:34.492 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:34.492 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:20:34.492 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.492 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:20:34.493 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:20:34.493 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:20:34.493 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:34.493 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:20:34.494 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:20:34.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:34.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:34.496 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:34.578 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:34.579 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:34.870 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:20:34.870 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.871 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-29T16:20:34.871 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:52.657461Z" 2020-11-29T16:20:34.871 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.871 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.872 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-29T16:20:34.872 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:44.658895Z" 2020-11-29T16:20:34.872 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.872 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.872 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-29T16:20:34.873 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:52.638714Z" 2020-11-29T16:20:34.873 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:34.873 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:34.873 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008081", 2020-11-29T16:20:34.873 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:09.181523Z" 2020-11-29T16:20:34.874 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:34.874 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:20:34.875 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:20:34.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:34.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:34.877 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:34.919 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:34.919 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:35.293 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:20:35.293 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:20:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:20:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:20:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:20:35.294 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 16089441476439148043 2020-11-29T16:20:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:20:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:35.295 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:20:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000012461", 2020-11-29T16:20:35.296 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:52.657461Z" 2020-11-29T16:20:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:35.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:20:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:35.298 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000013259", 2020-11-29T16:20:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:35.299 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:44.658895Z" 2020-11-29T16:20:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:35.300 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:20:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:35.301 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008761", 2020-11-29T16:20:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:35.302 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:52.638714Z" 2020-11-29T16:20:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:20:35.303 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:35.304 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008081", 2020-11-29T16:20:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:09.181523Z" 2020-11-29T16:20:35.305 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:20:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:35.306 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:20:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:20:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:20:35.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:35.307 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:20:35.308 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:20:35.309 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:35.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:35.310 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:35.388 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:35.388 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:35.658 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:20:35.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:35.659 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006751", 2020-11-29T16:20:35.659 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:52.643668Z" 2020-11-29T16:20:35.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:35.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:35.660 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:20:35.660 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.015365Z" 2020-11-29T16:20:35.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:35.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:35.661 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002770", 2020-11-29T16:20:35.661 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:52.656074Z" 2020-11-29T16:20:35.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:35.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:35.662 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004230", 2020-11-29T16:20:35.662 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:52.576814Z" 2020-11-29T16:20:35.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:35.662 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:20:35.663 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:20:35.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:35.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:35.665 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:35.706 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:35.707 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:35.989 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:20:35.990 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:20:35.990 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:20:35.991 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:20:35.991 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:20:35.991 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 201246450793848048 2020-11-29T16:20:35.992 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:35.992 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:20:35.993 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:35.993 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:20:35.994 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:35.994 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:35.994 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006751", 2020-11-29T16:20:35.995 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:35.995 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:35.996 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:35.996 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:52.643668Z" 2020-11-29T16:20:35.996 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:35.997 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:35.997 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:35.997 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:20:35.998 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:35.998 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:35.998 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:20:35.999 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:35.999 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:35.999 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:36.000 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.015365Z" 2020-11-29T16:20:36.000 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.000 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.001 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.001 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:20:36.001 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:36.001 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:36.001 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002770", 2020-11-29T16:20:36.002 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:36.002 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:36.002 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:36.002 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:52.656074Z" 2020-11-29T16:20:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:20:36.003 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000004230", 2020-11-29T16:20:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:36.004 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:52.576814Z" 2020-11-29T16:20:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:20:36.005 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:20:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:20:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:20:36.006 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.007 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:20:36.008 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:20:36.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:36.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:36.009 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:36.048 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:36.048 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:36.321 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:20:36.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:36.322 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006751", 2020-11-29T16:20:36.322 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:52.643668Z" 2020-11-29T16:20:36.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:36.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:36.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:20:36.323 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:00.015365Z" 2020-11-29T16:20:36.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:36.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:36.324 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002770", 2020-11-29T16:20:36.324 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:52.656074Z" 2020-11-29T16:20:36.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:36.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:36.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004230", 2020-11-29T16:20:36.325 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:19:52.576814Z" 2020-11-29T16:20:36.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:36.325 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:20:36.326 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:20:36.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:36.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:36.328 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:36.367 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:36.368 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:36.624 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:20:36.625 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:20:36.625 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:20:36.625 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:20:36.626 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:20:36.626 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 16876182154419717581 2020-11-29T16:20:36.626 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.626 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:20:36.627 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.627 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:20:36.627 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:36.627 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:36.628 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006751", 2020-11-29T16:20:36.628 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:36.628 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:36.628 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:36.629 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:52.643668Z" 2020-11-29T16:20:36.629 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.629 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.629 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.629 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:20:36.630 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:36.630 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:36.630 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006487", 2020-11-29T16:20:36.631 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:36.631 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:36.631 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:36.631 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:00.015365Z" 2020-11-29T16:20:36.632 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.632 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.632 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.633 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:20:36.633 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:36.633 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:36.633 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000002770", 2020-11-29T16:20:36.634 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:36.634 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:36.634 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:36.634 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:52.656074Z" 2020-11-29T16:20:36.635 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.635 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.635 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.635 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:20:36.635 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:36.636 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:36.636 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000004230", 2020-11-29T16:20:36.636 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:36.636 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:36.637 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:36.637 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:52.576814Z" 2020-11-29T16:20:36.637 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.637 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.637 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:20:36.638 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.638 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:20:36.638 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:20:36.638 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:20:36.639 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.639 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:20:36.640 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:20:36.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:36.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:36.642 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:36.721 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:36.722 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:36.990 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:20:36.990 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.991 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007815", 2020-11-29T16:20:36.991 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:08.796124Z" 2020-11-29T16:20:36.991 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.991 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.991 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008880", 2020-11-29T16:20:36.992 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:46.782553Z" 2020-11-29T16:20:36.992 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.992 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.992 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000004505", 2020-11-29T16:20:36.992 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:08.800630Z" 2020-11-29T16:20:36.993 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:36.993 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:36.993 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000004895", 2020-11-29T16:20:36.993 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:08.804785Z" 2020-11-29T16:20:36.994 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:36.994 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:20:36.995 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:20:36.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:36.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:36.996 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:37.080 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:37.080 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:37.376 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:20:37.377 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:20:37.377 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:20:37.377 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:20:37.377 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:20:37.378 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 3114293326551352523 2020-11-29T16:20:37.378 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:37.378 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:20:37.378 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:37.379 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:20:37.379 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:37.379 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:37.379 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007815", 2020-11-29T16:20:37.380 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:37.380 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:37.380 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:37.380 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:20:08.796124Z" 2020-11-29T16:20:37.380 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:37.381 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:37.381 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:37.381 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:20:37.381 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:37.382 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:37.382 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008880", 2020-11-29T16:20:37.382 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:37.382 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:37.383 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:37.383 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:19:46.782553Z" 2020-11-29T16:20:37.383 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:37.383 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:37.384 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:37.384 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:20:37.384 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:37.384 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:37.384 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000004505", 2020-11-29T16:20:37.385 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:37.385 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:37.385 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:37.385 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:20:08.800630Z" 2020-11-29T16:20:37.385 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:37.386 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:37.386 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:37.387 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:20:37.387 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:37.388 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:20:37.388 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000004895", 2020-11-29T16:20:37.388 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:37.389 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:37.389 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:37.390 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:20:08.804785Z" 2020-11-29T16:20:37.390 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:37.390 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:37.391 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:20:37.391 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:37.392 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:20:37.392 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:20:37.392 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:20:37.393 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:37.393 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:20:37.394 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:20:37.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:37.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:37.396 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:37.473 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:37.474 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:37.745 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:20:37.746 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:37.746 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007815", 2020-11-29T16:20:37.746 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:08.796124Z" 2020-11-29T16:20:37.746 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:37.747 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:37.747 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008880", 2020-11-29T16:20:37.747 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:19:46.782553Z" 2020-11-29T16:20:37.747 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:37.747 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:37.748 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000004505", 2020-11-29T16:20:37.748 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:08.800630Z" 2020-11-29T16:20:37.748 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:37.748 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:37.749 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000004895", 2020-11-29T16:20:37.749 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:08.804785Z" 2020-11-29T16:20:37.749 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:37.749 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:20:37.750 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:20:37.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:37.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:37.752 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:37.795 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:37.795 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:38.072 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:20:38.072 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:20:38.072 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:20:38.073 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:20:38.073 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:20:38.073 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 2748347558345283993 2020-11-29T16:20:38.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:38.073 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:20:38.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:38.074 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:20:38.074 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:38.074 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:38.075 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000007815", 2020-11-29T16:20:38.075 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:38.075 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:38.075 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:38.075 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:08.796124Z" 2020-11-29T16:20:38.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:38.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:38.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:38.076 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:20:38.077 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:38.077 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:38.077 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008880", 2020-11-29T16:20:38.077 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:38.077 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:38.078 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:38.078 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:19:46.782553Z" 2020-11-29T16:20:38.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:38.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:38.078 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:38.079 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:20:38.079 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:38.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:38.079 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004505", 2020-11-29T16:20:38.080 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:38.080 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:38.080 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:38.080 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:08.800630Z" 2020-11-29T16:20:38.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:38.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:38.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:38.081 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:20:38.081 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:38.081 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:20:38.082 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000004895", 2020-11-29T16:20:38.082 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:38.082 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:38.082 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:38.082 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:08.804785Z" 2020-11-29T16:20:38.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:38.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:38.083 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:20:38.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:38.084 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:20:38.084 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:20:38.084 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:20:38.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:38.084 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:20:38.085 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:20:38.086 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2020-11-29T16:20:38.087 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-62 2020-11-29T16:20:38.128 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-63 2020-11-29T16:20:38.203 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-64 2020-11-29T16:20:38.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:38.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:38.859 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:38.912 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:38.913 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:39.160 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:20:39.217 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:20:39.217 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:39.217 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:20:39.217 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:20:39.217 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:39.218 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:39.218 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666838.244530_414.1", 2020-11-29T16:20:39.218 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:38.244530Z" 2020-11-29T16:20:39.218 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:39.218 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:39.219 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:20:39.219 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:20:39.219 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:39.219 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:39.219 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:20:39.220 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:20:39.220 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:39.220 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:20:39.221 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:20:39.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:39.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:39.222 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:39.263 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:39.263 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:39.521 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:20:39.521 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:20:39.521 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:20:39.522 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:20:39.522 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:20:39.522 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:20:39.522 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:20:39.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:39.523 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:20:39.523 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:39.523 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:20:39.523 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:39.523 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:20:39.524 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:20:39.524 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:39.524 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:20:39.524 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:39.524 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:20:39.525 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:20:39.525 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:39.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:39.525 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:39.526 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:20:39.526 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:39.526 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:20:39.526 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666838.244530_414.1", 2020-11-29T16:20:39.526 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:39.527 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:39.527 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:39.527 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:38.244530Z", 2020-11-29T16:20:39.527 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:20:39.528 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:39.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:39.528 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:39.528 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:20:39.528 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:39.529 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:20:39.529 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:20:39.529 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:39.529 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:39.530 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:39.530 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:20:39.530 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:20:39.530 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:39.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:39.531 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:39.531 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:20:39.531 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:39.531 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:20:39.532 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:20:39.532 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:39.532 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:39.532 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:39.532 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:20:39.533 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:20:39.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:39.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:39.533 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:20:39.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:39.534 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:20:39.534 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:20:39.534 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:20:39.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:39.534 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:20:39.535 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:20:39.536 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:20:39.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:39.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:39.537 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:39.577 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:39.577 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:39.856 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:20:39.857 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:20:39.857 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:20:39.857 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:20:39.857 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:20:39.858 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:20:39.858 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:20:39.858 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:39.858 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:20:39.859 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:39.859 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:20:39.859 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:39.859 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:20:39.859 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:20:39.860 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:39.860 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:20:39.860 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:39.860 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:20:39.860 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:20:39.861 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:39.861 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:39.861 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:39.861 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:20:39.861 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:39.862 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:20:39.862 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666838.244530_414.1", 2020-11-29T16:20:39.862 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:39.862 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:39.863 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:39.863 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:20:38.244530Z", 2020-11-29T16:20:39.863 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:20:39.863 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:39.863 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:39.864 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:39.864 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:20:39.864 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:39.864 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:20:39.864 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:20:39.865 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:39.865 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:39.865 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:39.865 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:20:39.865 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:20:39.866 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:39.866 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:39.866 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:39.866 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:20:39.866 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:20:39.867 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:20:39.867 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:20:39.867 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:20:39.867 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:20:39.867 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:20:39.868 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:20:39.868 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:20:39.868 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:39.868 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:39.868 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:20:39.869 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:39.869 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:20:39.869 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:20:39.869 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:20:39.869 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:39.870 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:20:39.870 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:20:39.871 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:20:39.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:39.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:39.872 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:39.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:39.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:40.182 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:20:40.182 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:20:40.183 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:20:40.183 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:20:40.184 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:20:40.184 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:20:40.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:20:40.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:40.185 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:20:40.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:40.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:20:40.187 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:40.187 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:20:40.187 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:20:40.188 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:40.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:20:40.189 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:40.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:20:40.189 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:20:40.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:40.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:40.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:40.191 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:20:40.191 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:40.192 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:20:40.192 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666838.244530_414.1", 2020-11-29T16:20:40.192 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:40.193 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:40.193 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:38.244530Z", 2020-11-29T16:20:40.194 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:20:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:40.195 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:20:40.196 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:20:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:20:40.197 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:40.198 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:20:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:20:40.199 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:40.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:40.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:20:40.200 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:20:40.201 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:20:40.201 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:20:40.202 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:20:40.202 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:20:40.203 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:20:40.203 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:20:40.203 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:20:40.204 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:20:40.204 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:40.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:40.205 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:20:40.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:20:40.206 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:20:40.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:20:40.207 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:20:40.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:20:40.207 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:20:40.209 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:20:40.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:40.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:40.211 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:40.259 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:40.259 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:40.523 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:40.513+0000 7f712f1fc080 1 bucket sync caught up with source: 2020-11-29T16:20:40.523 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.49.5, , 00000000001.47.5, , 00000000001.66.5, , , 00000000001.61.5, , , ] 2020-11-29T16:20:40.523 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.49.5, , 00000000001.47.5, , 00000000001.66.5, , , 00000000001.61.5, , , ] 2020-11-29T16:20:40.523 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:40.513+0000 7f712f1fc080 0 bucket checkpoint complete 2020-11-29T16:20:40.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:40.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:40.559 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:40.607 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:40.607 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:40.861 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:40.855+0000 7f11e7953080 1 bucket sync caught up with source: 2020-11-29T16:20:40.861 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.49.5, , 00000000001.47.5, , 00000000001.66.5, , , 00000000001.61.5, , , ] 2020-11-29T16:20:40.861 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.49.5, , 00000000001.47.5, , 00000000001.66.5, , , 00000000001.61.5, , , ] 2020-11-29T16:20:40.861 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:40.855+0000 7f11e7953080 0 bucket checkpoint complete 2020-11-29T16:20:40.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:40.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:40.890 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:40.937 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:40.937 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:41.197+0000 7f74d943d080 1 bucket sync caught up with source: 2020-11-29T16:20:41.205 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.49.5, , 00000000001.47.5, , 00000000001.66.5, , , 00000000001.61.5, , , ] 2020-11-29T16:20:41.206 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.49.5, , 00000000001.47.5, , 00000000001.66.5, , , 00000000001.61.5, , , ] 2020-11-29T16:20:41.206 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:41.197+0000 7f74d943d080 0 bucket checkpoint complete 2020-11-29T16:20:41.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:41.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:41.309 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:41.363 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:41.363 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:41.607 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:41.599+0000 7f0519d3f080 1 bucket sync caught up with source: 2020-11-29T16:20:41.607 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.49.5, , 00000000001.48.5, , 00000000001.55.5, , , 00000000001.64.5, , , ] 2020-11-29T16:20:41.608 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.49.5, , 00000000001.48.5, , 00000000001.55.5, , , 00000000001.64.5, , , ] 2020-11-29T16:20:41.608 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:41.599+0000 7f0519d3f080 0 bucket checkpoint complete 2020-11-29T16:20:41.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:41.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:41.641 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:41.688 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:41.689 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:41.970 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:41.963+0000 7fceb8bac080 1 bucket sync caught up with source: 2020-11-29T16:20:41.970 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.49.5, , 00000000001.48.5, , 00000000001.55.5, , , 00000000001.64.5, , , ] 2020-11-29T16:20:41.971 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.49.5, , 00000000001.48.5, , 00000000001.55.5, , , 00000000001.64.5, , , ] 2020-11-29T16:20:41.971 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:41.963+0000 7fceb8bac080 0 bucket checkpoint complete 2020-11-29T16:20:42.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:42.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:42.007 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:42.053 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:42.054 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:42.320 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:42.315+0000 7f5feb05c080 1 bucket sync caught up with source: 2020-11-29T16:20:42.320 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.45.5, , 00000000001.47.5, , 00000000001.56.5, , , 00000000001.72.5, , , ] 2020-11-29T16:20:42.320 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.45.5, , 00000000001.47.5, , 00000000001.56.5, , , 00000000001.72.5, , , ] 2020-11-29T16:20:42.321 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:42.315+0000 7f5feb05c080 0 bucket checkpoint complete 2020-11-29T16:20:42.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:42.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:42.352 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:42.399 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:42.400 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:42.660 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:42.653+0000 7fa4a5482080 1 bucket sync caught up with source: 2020-11-29T16:20:42.660 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.45.5, , 00000000001.47.5, , 00000000001.56.5, , , 00000000001.72.5, , , ] 2020-11-29T16:20:42.661 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.45.5, , 00000000001.47.5, , 00000000001.56.5, , , 00000000001.72.5, , , ] 2020-11-29T16:20:42.661 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:42.653+0000 7fa4a5482080 0 bucket checkpoint complete 2020-11-29T16:20:42.694 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-62 zones={test-zone2, test-zone1} 2020-11-29T16:20:42.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:42.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:42.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:42.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:42.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:42.713 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:42.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:42.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:42.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:42.764 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:42.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:42.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:42.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:43.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:43.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:43.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:43.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:43.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:43.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:43.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:43.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:43.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:43.862 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-62 zones={test-zone2, test-zone1} 2020-11-29T16:20:43.865 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-62 zones={test-zone3, test-zone1} 2020-11-29T16:20:43.875 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:43.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:43.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:43.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:43.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:43.882 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:43.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:43.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:43.888 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:43.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:43.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:43.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:44.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:44.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:44.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:44.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:44.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:44.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:44.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:44.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:44.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:44.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:45.007 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-62 zones={test-zone3, test-zone1} 2020-11-29T16:20:45.008 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-62 zones={test-zone3, test-zone2} 2020-11-29T16:20:45.017 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:45.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:45.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:45.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:45.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:45.024 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:45.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:45.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:45.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:45.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:45.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:45.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:45.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:45.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:45.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:45.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:45.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:45.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:45.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:45.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:45.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:46.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:46.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-62 zones={test-zone3, test-zone2} 2020-11-29T16:20:46.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:46.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:46.167 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:46.212 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:46.212 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:46.453 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:46.445+0000 7fd473248080 1 bucket sync caught up with source: 2020-11-29T16:20:46.453 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.68.5, , 00000000001.46.5, , 00000000001.37.5, , , 00000000002.43.5, , , ] 2020-11-29T16:20:46.454 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.68.5, , 00000000001.46.5, , 00000000001.37.5, , , 00000000002.43.5, , , ] 2020-11-29T16:20:46.454 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:46.445+0000 7fd473248080 0 bucket checkpoint complete 2020-11-29T16:20:46.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:46.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:46.484 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:46.531 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:46.532 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:46.799 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:46.790+0000 7f6bdacfa080 1 bucket sync caught up with source: 2020-11-29T16:20:46.799 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.68.5, , 00000000001.46.5, , 00000000001.37.5, , , 00000000002.43.5, , , ] 2020-11-29T16:20:46.800 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.68.5, , 00000000001.46.5, , 00000000001.37.5, , , 00000000002.43.5, , , ] 2020-11-29T16:20:46.800 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:46.790+0000 7f6bdacfa080 0 bucket checkpoint complete 2020-11-29T16:20:46.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:46.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:46.827 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:46.876 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:46.877 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:47.111 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:47.105+0000 7f4be3989080 1 bucket sync caught up with source: 2020-11-29T16:20:47.112 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.68.5, , 00000000001.46.5, , 00000000001.37.5, , , 00000000002.43.5, , , ] 2020-11-29T16:20:47.112 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.68.5, , 00000000001.46.5, , 00000000001.37.5, , , 00000000002.43.5, , , ] 2020-11-29T16:20:47.112 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:47.105+0000 7f4be3989080 0 bucket checkpoint complete 2020-11-29T16:20:47.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:47.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:47.244 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:47.293 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:47.293 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:47.561 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:47.554+0000 7fb26a16c080 1 bucket sync caught up with source: 2020-11-29T16:20:47.561 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.57.5, , 00000000001.50.5, , 00000000001.38.5, , , 00000000001.48.5, , , ] 2020-11-29T16:20:47.561 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.57.5, , 00000000001.50.5, , 00000000001.38.5, , , 00000000001.48.5, , , ] 2020-11-29T16:20:47.561 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:47.554+0000 7fb26a16c080 0 bucket checkpoint complete 2020-11-29T16:20:47.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:47.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:47.598 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:47.645 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:47.645 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:47.903 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:47.898+0000 7efefecf8080 1 bucket sync caught up with source: 2020-11-29T16:20:47.904 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.57.5, , 00000000001.50.5, , 00000000001.38.5, , , 00000000001.48.5, , , ] 2020-11-29T16:20:47.904 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.57.5, , 00000000001.50.5, , 00000000001.38.5, , , 00000000001.48.5, , , ] 2020-11-29T16:20:47.904 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:47.898+0000 7efefecf8080 0 bucket checkpoint complete 2020-11-29T16:20:47.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:47.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:47.935 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:47.980 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:47.981 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:48.242 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:48.234+0000 7fb8c9b52080 1 bucket sync caught up with source: 2020-11-29T16:20:48.243 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.58.5, , 00000000001.43.5, , 00000000001.40.5, , , 00000000001.42.5, , , ] 2020-11-29T16:20:48.243 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.58.5, , 00000000001.43.5, , 00000000001.40.5, , , 00000000001.42.5, , , ] 2020-11-29T16:20:48.243 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:48.234+0000 7fb8c9b52080 0 bucket checkpoint complete 2020-11-29T16:20:48.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:48.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:48.273 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:48.320 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:48.320 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:48.590 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:48.581+0000 7ff7e2530080 1 bucket sync caught up with source: 2020-11-29T16:20:48.590 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.58.5, , 00000000001.43.5, , 00000000001.40.5, , , 00000000001.42.5, , , ] 2020-11-29T16:20:48.591 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.58.5, , 00000000001.43.5, , 00000000001.40.5, , , 00000000001.42.5, , , ] 2020-11-29T16:20:48.591 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:48.581+0000 7ff7e2530080 0 bucket checkpoint complete 2020-11-29T16:20:48.622 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-63 zones={test-zone2, test-zone1} 2020-11-29T16:20:48.633 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:48.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:48.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:48.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:48.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:48.641 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:48.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:48.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:48.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:48.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:48.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:48.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:48.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:48.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:48.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:49.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:49.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:49.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:49.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:49.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:49.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:49.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:49.745 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-63 zones={test-zone2, test-zone1} 2020-11-29T16:20:49.745 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-63 zones={test-zone3, test-zone1} 2020-11-29T16:20:49.754 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:49.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:49.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:49.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:49.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:49.760 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:49.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:49.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:49.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:49.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:49.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:49.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:49.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:50.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:50.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:50.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:50.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:50.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:50.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:50.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:50.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:50.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:50.857 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-63 zones={test-zone3, test-zone1} 2020-11-29T16:20:50.858 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-63 zones={test-zone3, test-zone2} 2020-11-29T16:20:50.870 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:50.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:50.876 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:50.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:50.877 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:50.877 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:50.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:50.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:50.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:50.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:50.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:50.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:51.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:51.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:51.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:51.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:51.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:51.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:51.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:51.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:51.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:51.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:51.997 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-63 zones={test-zone3, test-zone2} 2020-11-29T16:20:51.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:51.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:51.999 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:52.042 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:52.043 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:52.326 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:52.317+0000 7f625887d080 1 bucket sync caught up with source: 2020-11-29T16:20:52.326 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.66.5, , 00000000001.64.5, , 00000000001.61.5, , , 00000000001.63.5, , , ] 2020-11-29T16:20:52.327 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.66.5, , 00000000001.64.5, , 00000000001.61.5, , , 00000000001.63.5, , , ] 2020-11-29T16:20:52.327 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:52.317+0000 7f625887d080 0 bucket checkpoint complete 2020-11-29T16:20:52.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:52.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:52.361 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:52.404 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:52.404 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:52.643 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:52.638+0000 7f7217e90080 1 bucket sync caught up with source: 2020-11-29T16:20:52.644 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.66.5, , 00000000001.64.5, , 00000000001.61.5, , , 00000000001.63.5, , , ] 2020-11-29T16:20:52.644 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.66.5, , 00000000001.64.5, , 00000000001.61.5, , , 00000000001.63.5, , , ] 2020-11-29T16:20:52.644 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:52.638+0000 7f7217e90080 0 bucket checkpoint complete 2020-11-29T16:20:52.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:52.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:52.675 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:52.723 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:52.723 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:52.990 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:52.980+0000 7feb59197080 1 bucket sync caught up with source: 2020-11-29T16:20:52.990 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.66.5, , 00000000001.64.5, , 00000000001.61.5, , , 00000000001.63.5, , , ] 2020-11-29T16:20:52.990 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.66.5, , 00000000001.64.5, , 00000000001.61.5, , , 00000000001.63.5, , , ] 2020-11-29T16:20:52.991 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:52.980+0000 7feb59197080 0 bucket checkpoint complete 2020-11-29T16:20:53.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:53.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:53.097 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:53.146 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:53.147 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:53.398 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:53.390+0000 7faf21b88080 1 bucket sync caught up with source: 2020-11-29T16:20:53.398 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.68.5, , 00000000001.67.5, , 00000000001.76.5, , , 00000000001.66.5, , , ] 2020-11-29T16:20:53.398 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.68.5, , 00000000001.67.5, , 00000000001.76.5, , , 00000000001.66.5, , , ] 2020-11-29T16:20:53.399 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:53.390+0000 7faf21b88080 0 bucket checkpoint complete 2020-11-29T16:20:53.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:53.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:53.433 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:53.478 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:53.478 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:53.743 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:53.738+0000 7f52c697a080 1 bucket sync caught up with source: 2020-11-29T16:20:53.744 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.68.5, , 00000000001.67.5, , 00000000001.76.5, , , 00000000001.66.5, , , ] 2020-11-29T16:20:53.744 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.68.5, , 00000000001.67.5, , 00000000001.76.5, , , 00000000001.66.5, , , ] 2020-11-29T16:20:53.744 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:53.738+0000 7f52c697a080 0 bucket checkpoint complete 2020-11-29T16:20:53.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:53.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:53.778 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:53.865 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:53.865 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:54.139 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:54.130+0000 7f83f0a36080 1 bucket sync caught up with source: 2020-11-29T16:20:54.139 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000001.77.5, , 00000000001.75.5, , 00000000001.76.5, , , 00000000001.74.5, , , ] 2020-11-29T16:20:54.140 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000001.77.5, , 00000000001.75.5, , 00000000001.76.5, , , 00000000001.74.5, , , ] 2020-11-29T16:20:54.140 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:20:54.130+0000 7f83f0a36080 0 bucket checkpoint complete 2020-11-29T16:20:54.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:54.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:20:54.171 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:20:54.218 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:54.219 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:54.493 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:54.484+0000 7fc21761b080 1 bucket sync caught up with source: 2020-11-29T16:20:54.494 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.77.5, , 00000000001.75.5, , 00000000001.76.5, , , 00000000001.74.5, , , ] 2020-11-29T16:20:54.494 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.77.5, , 00000000001.75.5, , 00000000001.76.5, , , 00000000001.74.5, , , ] 2020-11-29T16:20:54.494 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:20:54.484+0000 7fc21761b080 0 bucket checkpoint complete 2020-11-29T16:20:54.527 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-64 zones={test-zone2, test-zone1} 2020-11-29T16:20:54.537 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:54.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:54.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:54.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:54.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:54.588 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:54.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:54.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:54.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:54.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:54.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:54.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:54.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:54.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:54.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:55.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:55.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:55.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:55.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:55.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:55.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:55.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:55.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-64 zones={test-zone2, test-zone1} 2020-11-29T16:20:55.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-64 zones={test-zone3, test-zone1} 2020-11-29T16:20:55.771 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:55.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:55.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:55.777 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:55.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:55.778 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:55.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:55.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:55.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:55.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:55.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:55.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:55.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:56.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:56.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:56.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:56.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:56.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:56.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:56.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:56.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:56.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:56.881 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-64 zones={test-zone3, test-zone1} 2020-11-29T16:20:56.882 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-64 zones={test-zone3, test-zone2} 2020-11-29T16:20:56.892 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:20:56.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:56.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:56.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:56.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:56.900 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:20:56.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:20:56.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:20:56.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:20:56.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:20:56.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:56.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:57.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:20:57.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:57.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:57.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:20:57.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:57.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:57.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:20:57.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:57.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:57.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:20:58.061 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-64 zones={test-zone3, test-zone2} 2020-11-29T16:20:58.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:20:58.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:20:58.063 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:20:58.107 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:58.107 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:58.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:20:58.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:20:58.444 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:20:58.525 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:58.525 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:58.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'kcmyep-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:20:58.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'kcmyep-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:20:58.881 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket kcmyep-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:20:58.963 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:58.964 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:59.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:59.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:59.299 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:59.376 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:59.377 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:20:59.617 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:20:59.652 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:20:59.652 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:59.653 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:20:59.653 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:20:59.653 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:59.653 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:59.654 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666859.241004_429.1", 2020-11-29T16:20:59.654 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:20:59.241004Z" 2020-11-29T16:20:59.654 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:59.654 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:59.654 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:20:59.655 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:20:59.655 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:20:59.655 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:20:59.655 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:20:59.656 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:20:59.656 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:20:59.656 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:20:59.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:20:59.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:59.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:20:59.659 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:20:59.701 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:20:59.702 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:00.010 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:21:00.010 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:21:00.011 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:21:00.011 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:21:00.011 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:21:00.011 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:21:00.011 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:21:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:21:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:00.012 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:21:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:21:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:00.013 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:21:00.014 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:00.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:21:00.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:00.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:00.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:00.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:00.015 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:21:00.015 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:00.015 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:00.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666858.388517_424.1", 2020-11-29T16:21:00.016 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:00.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:00.016 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:00.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:58.388517Z", 2020-11-29T16:21:00.017 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:21:00.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:00.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:00.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:00.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:21:00.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:00.018 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:00.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:21:00.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:00.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:00.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:00.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:21:00.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:00.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:00.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:00.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:00.020 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:21:00.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:00.020 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:00.021 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:21:00.021 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:00.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:00.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:00.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:21:00.022 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:00.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:00.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:00.022 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:21:00.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:00.023 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:21:00.023 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:21:00.023 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:21:00.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:00.023 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:21:00.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1606666859.241004_429.1 target=1_1606666858.388517_424.1 2020-11-29T16:21:05.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:21:05.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:21:05.027 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:21:05.068 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:05.068 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:05.340 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:21:05.340 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:21:05.341 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:21:05.341 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:21:05.341 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:21:05.341 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:21:05.342 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:21:05.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:05.342 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:21:05.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:05.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:21:05.343 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:05.343 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:05.343 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:21:05.343 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:05.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:21:05.344 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:05.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:21:05.344 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:05.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:05.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:05.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:05.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:21:05.345 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:05.345 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:05.346 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666859.241004_429.1", 2020-11-29T16:21:05.346 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:05.346 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:05.346 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:05.346 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:59.241004Z", 2020-11-29T16:21:05.347 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:21:05.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:05.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:05.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:05.347 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:21:05.348 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:05.348 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:05.348 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:21:05.348 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:05.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:05.349 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:05.349 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:21:05.349 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:05.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:05.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:05.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:05.350 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:21:05.350 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:05.350 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:05.350 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:21:05.351 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:05.351 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:05.351 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:05.351 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:21:05.351 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:05.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:05.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:05.352 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:21:05.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:05.353 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:21:05.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:21:05.353 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:21:05.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:05.354 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:21:05.355 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:21:05.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:21:05.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:21:05.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:21:05.357 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:21:05.396 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:05.396 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:05.690 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:21:05.690 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:21:05.691 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:21:05.691 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:21:05.691 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:21:05.691 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:21:05.691 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:21:05.692 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:21:05.692 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:21:05.692 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:21:05.693 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:21:05.693 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:21:05.693 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:21:05.693 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:21:05.694 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:21:05.694 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:21:05.694 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:21:05.695 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:21:05.695 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:21:05.695 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:21:05.695 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:21:05.695 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:21:05.696 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:21:05.696 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:21:05.696 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:21:05.696 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606666859.241004_429.1", 2020-11-29T16:21:05.697 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:21:05.697 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:21:05.697 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:21:05.697 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:20:59.241004Z", 2020-11-29T16:21:05.698 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:21:05.698 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:21:05.698 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:21:05.698 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:21:05.699 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:21:05.699 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:21:05.699 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:21:05.699 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:21:05.699 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:21:05.700 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:21:05.700 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:21:05.700 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:21:05.700 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:21:05.700 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:21:05.701 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:21:05.701 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:21:05.701 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:21:05.701 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:21:05.702 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:21:05.702 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:21:05.702 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:21:05.702 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:21:05.703 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:21:05.703 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:21:05.703 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:21:05.703 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:21:05.704 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:21:05.704 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:21:05.704 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:21:05.704 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:21:05.704 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:21:05.705 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:21:05.705 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:21:05.705 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:21:05.706 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:21:05.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:21:05.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:21:05.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:21:05.708 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:21:05.749 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:05.750 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:06.024 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:21:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:21:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:21:06.025 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:21:06.026 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:21:06.026 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:21:06.026 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:21:06.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:06.027 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:21:06.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:06.027 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:21:06.027 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:06.028 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:06.028 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:21:06.028 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:06.028 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:21:06.028 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:06.029 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:21:06.029 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:06.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:06.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:06.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:06.030 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:21:06.030 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:06.030 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:06.030 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606666859.241004_429.1", 2020-11-29T16:21:06.031 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:06.031 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:06.031 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:06.031 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:20:59.241004Z", 2020-11-29T16:21:06.032 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:21:06.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:06.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:06.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:06.033 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:21:06.033 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:06.033 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:06.034 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:21:06.034 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:06.034 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:06.034 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:06.035 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:21:06.035 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:06.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:06.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:06.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:21:06.036 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:21:06.036 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:21:06.036 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:21:06.036 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:21:06.037 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:21:06.037 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:21:06.037 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:21:06.038 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:21:06.038 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:21:06.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:06.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:06.038 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:21:06.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:21:06.039 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:21:06.039 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:21:06.039 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:21:06.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:21:06.040 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:21:06.041 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:21:06.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kcmyep-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:21:06.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kcmyep-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:21:06.635 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kcmyep-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:21:06.675 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:06.675 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:07.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kcmyep-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:21:07.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kcmyep-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:21:07.043 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kcmyep-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:21:07.124 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:07.125 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:07.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'kcmyep-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:21:07.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'kcmyep-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:21:07.460 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket kcmyep-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:21:07.541 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:07.541 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:07.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:07.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:07.898 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:07.947 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:07.947 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:08.218 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:08.212+0000 7f829749d080 1 waiting for incremental sync to catch up: 2020-11-29T16:21:08.218 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:21:08.218 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:13.222 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:13.216+0000 7f829749d080 1 bucket sync caught up with source: 2020-11-29T16:21:13.223 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:13.223 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:13.223 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:13.216+0000 7f829749d080 0 bucket checkpoint complete 2020-11-29T16:21:13.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:13.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:13.256 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:13.305 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:13.306 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:13.557 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:13.550+0000 7fc35a39e080 1 waiting for incremental sync to catch up: 2020-11-29T16:21:13.558 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.60.1, , 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, , 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, , 00000000004.52.1] 2020-11-29T16:21:13.558 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:18.561 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:18.554+0000 7fc35a39e080 1 waiting for incremental sync to catch up: 2020-11-29T16:21:18.562 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.60.1, , 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, , 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, , 00000000004.52.1] 2020-11-29T16:21:18.562 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:23.566 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:23.558+0000 7fc35a39e080 1 bucket sync caught up with source: 2020-11-29T16:21:23.566 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:23.567 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:23.567 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:23.558+0000 7fc35a39e080 0 bucket checkpoint complete 2020-11-29T16:21:23.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:23.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:23.599 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:23.647 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:23.647 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:23.890 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:23.884+0000 7f07f0f47080 1 bucket sync caught up with source: 2020-11-29T16:21:23.891 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:23.891 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.60.1, 00000000004.85.1, 00000000006.61.1, 00000000004.43.1, 00000000006.76.1, 00000000004.43.1, 00000000004.50.1, 00000000006.71.1, 00000000004.53.1, 00000000004.166.1, 00000000004.52.1] 2020-11-29T16:21:23.891 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:23.884+0000 7f07f0f47080 0 bucket checkpoint complete 2020-11-29T16:21:24.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:24.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:24.023 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:24.075 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:24.076 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:24.346 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:24.338+0000 7f54ff0b9080 1 bucket sync caught up with source: 2020-11-29T16:21:24.346 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.61.5, 00000000002.91.1, 00000000004.59.5, 00000000002.45.1, 00000000004.69.5, 00000000002.46.1, 00000000002.41.1, 00000000004.78.5, 00000000002.58.1, 00000000002.175.1, 00000000002.58.1] 2020-11-29T16:21:24.346 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.61.5, 00000000002.91.1, 00000000004.59.5, 00000000002.45.1, 00000000004.69.5, 00000000002.46.1, 00000000002.41.1, 00000000004.78.5, 00000000002.58.1, 00000000002.175.1, 00000000002.58.1] 2020-11-29T16:21:24.347 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:24.338+0000 7f54ff0b9080 0 bucket checkpoint complete 2020-11-29T16:21:24.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:24.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:24.374 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:24.419 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:24.420 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:24.671 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:24.662+0000 7fbe40681080 1 bucket sync caught up with source: 2020-11-29T16:21:24.671 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.61.5, 00000000002.91.1, 00000000004.59.5, 00000000002.45.1, 00000000004.69.5, 00000000002.46.1, 00000000002.41.1, 00000000004.78.5, 00000000002.58.1, 00000000002.175.1, 00000000002.58.1] 2020-11-29T16:21:24.671 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.61.5, 00000000002.91.1, 00000000004.59.5, 00000000002.45.1, 00000000004.69.5, 00000000002.46.1, 00000000002.41.1, 00000000004.78.5, 00000000002.58.1, 00000000002.175.1, 00000000002.58.1] 2020-11-29T16:21:24.671 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:24.662+0000 7fbe40681080 0 bucket checkpoint complete 2020-11-29T16:21:24.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:24.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:24.702 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:24.745 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:24.745 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:24.999 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:24.990+0000 7f1ae6ed0080 1 bucket sync caught up with source: 2020-11-29T16:21:24.999 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.59.5, 00000000002.105.1, 00000000004.57.5, 00000000002.43.1, 00000000004.68.5, 00000000002.46.1, 00000000002.49.1, 00000000004.86.5, 00000000002.52.1, 00000000002.175.1, 00000000002.49.1] 2020-11-29T16:21:25.000 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.59.5, 00000000002.105.1, 00000000004.57.5, 00000000002.43.1, 00000000004.68.5, 00000000002.46.1, 00000000002.49.1, 00000000004.86.5, 00000000002.52.1, 00000000002.175.1, 00000000002.49.1] 2020-11-29T16:21:25.000 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:24.990+0000 7f1ae6ed0080 0 bucket checkpoint complete 2020-11-29T16:21:25.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:25.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-62', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:25.028 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-62 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:25.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:25.076 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:25.345 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:25.340+0000 7fedf6324080 1 bucket sync caught up with source: 2020-11-29T16:21:25.345 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000004.59.5, 00000000002.105.1, 00000000004.57.5, 00000000002.43.1, 00000000004.68.5, 00000000002.46.1, 00000000002.49.1, 00000000004.86.5, 00000000002.52.1, 00000000002.175.1, 00000000002.49.1] 2020-11-29T16:21:25.345 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000004.59.5, 00000000002.105.1, 00000000004.57.5, 00000000002.43.1, 00000000004.68.5, 00000000002.46.1, 00000000002.49.1, 00000000004.86.5, 00000000002.52.1, 00000000002.175.1, 00000000002.49.1] 2020-11-29T16:21:25.345 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:25.340+0000 7fedf6324080 0 bucket checkpoint complete 2020-11-29T16:21:25.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-62 zones={test-zone2, test-zone1} 2020-11-29T16:21:25.389 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:25.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:25.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:25.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:25.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:25.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:25.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:25.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:25.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:25.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:25.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:25.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:25.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:25.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:25.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:25.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:25.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:25.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:25.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:25.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:25.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:25.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:25.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:25.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:25.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:26.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:26.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:26.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:26.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:26.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:26.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:26.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:26.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:26.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:26.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:26.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:27.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:27.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:27.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:27.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:27.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:27.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:27.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-62 zones={test-zone2, test-zone1} 2020-11-29T16:21:27.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-62 zones={test-zone3, test-zone1} 2020-11-29T16:21:27.643 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:27.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:27.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:27.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:27.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:27.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:27.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:27.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:27.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:27.651 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:27.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:27.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:27.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:27.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:27.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:27.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:27.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:27.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:27.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:27.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:27.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:27.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:28.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:28.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:28.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:28.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:28.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:28.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:28.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:28.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:28.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:28.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:28.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:29.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:29.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:29.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:29.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:29.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:29.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:29.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:29.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:29.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:29.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-62 zones={test-zone3, test-zone1} 2020-11-29T16:21:29.950 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-62 zones={test-zone3, test-zone2} 2020-11-29T16:21:29.962 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:29.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:29.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:29.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:29.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:29.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:29.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:29.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:29.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:29.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:29.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:29.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:29.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:29.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:29.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:29.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:29.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:29.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:29.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:30.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:30.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:30.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:30.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:30.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:30.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:30.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:30.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:30.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:30.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:30.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:31.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:31.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:31.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:31.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:31.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:31.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:31.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:31.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:31.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:31.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:31.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:32.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:32.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-62 zones={test-zone3, test-zone2} 2020-11-29T16:21:32.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:32.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:32.211 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:32.254 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:32.254 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:32.523 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:32.520+0000 7f541d464080 1 waiting for incremental sync to catch up: 2020-11-29T16:21:32.523 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000006.47.1, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:32.524 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000007.49.5, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:37.527 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:37.524+0000 7f541d464080 1 waiting for incremental sync to catch up: 2020-11-29T16:21:37.528 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000006.47.1, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:37.528 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000007.49.5, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:42.532 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:42.527+0000 7f541d464080 1 bucket sync caught up with source: 2020-11-29T16:21:42.533 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000007.49.5, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:42.533 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000007.49.5, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:42.533 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:42.527+0000 7f541d464080 0 bucket checkpoint complete 2020-11-29T16:21:42.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:42.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:42.566 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:42.613 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:42.613 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:42.866 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:42.858+0000 7f61a9497080 1 bucket sync caught up with source: 2020-11-29T16:21:42.866 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000007.49.5, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:42.866 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000007.49.5, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:42.866 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:42.858+0000 7f61a9497080 0 bucket checkpoint complete 2020-11-29T16:21:42.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:42.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:42.892 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:42.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:42.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:43.198 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:43.191+0000 7f25e4321080 1 bucket sync caught up with source: 2020-11-29T16:21:43.199 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000007.49.5, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:43.199 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.80.5, 00000000004.38.1, 00000000006.56.5, 00000000004.88.1, 00000000006.47.5, 00000000004.89.1, 00000000004.55.1, 00000000007.49.5, 00000000004.48.1, 00000000004.64.1, 00000000004.76.1] 2020-11-29T16:21:43.199 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:43.191+0000 7f25e4321080 0 bucket checkpoint complete 2020-11-29T16:21:43.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:43.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:43.416 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:43.464 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:43.464 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:43.710 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:43.702+0000 7f4b1e897080 1 bucket sync caught up with source: 2020-11-29T16:21:43.710 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.64.1, 00000000002.31.1, 00000000004.57.1, 00000000002.86.1, 00000000004.45.1, 00000000002.90.1, 00000000002.58.1, 00000000004.53.1, 00000000002.47.1, 00000000002.56.1, 00000000002.80.1] 2020-11-29T16:21:43.711 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.64.1, 00000000002.31.1, 00000000004.57.1, 00000000002.86.1, 00000000004.45.1, 00000000002.90.1, 00000000002.58.1, 00000000004.53.1, 00000000002.47.1, 00000000002.56.1, 00000000002.80.1] 2020-11-29T16:21:43.711 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:43.702+0000 7f4b1e897080 0 bucket checkpoint complete 2020-11-29T16:21:43.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:43.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:43.776 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:43.824 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:43.824 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:44.086 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:44.078+0000 7f1ac5983080 1 bucket sync caught up with source: 2020-11-29T16:21:44.087 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.64.1, 00000000002.31.1, 00000000004.57.1, 00000000002.86.1, 00000000004.45.1, 00000000002.90.1, 00000000002.58.1, 00000000004.53.1, 00000000002.47.1, 00000000002.56.1, 00000000002.80.1] 2020-11-29T16:21:44.087 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.64.1, 00000000002.31.1, 00000000004.57.1, 00000000002.86.1, 00000000004.45.1, 00000000002.90.1, 00000000002.58.1, 00000000004.53.1, 00000000002.47.1, 00000000002.56.1, 00000000002.80.1] 2020-11-29T16:21:44.087 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:44.078+0000 7f1ac5983080 0 bucket checkpoint complete 2020-11-29T16:21:44.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:44.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:44.121 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:44.167 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:44.167 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:44.419 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:44.414+0000 7f42d880f080 1 bucket sync caught up with source: 2020-11-29T16:21:44.420 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.70.5, 00000000002.30.1, 00000000004.53.5, 00000000002.101.1, 00000000004.50.5, 00000000002.104.1, 00000000002.58.1, 00000000004.48.5, 00000000002.57.1, 00000000002.54.1, 00000000002.90.1] 2020-11-29T16:21:44.420 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.70.5, 00000000002.30.1, 00000000004.53.5, 00000000002.101.1, 00000000004.50.5, 00000000002.104.1, 00000000002.58.1, 00000000004.48.5, 00000000002.57.1, 00000000002.54.1, 00000000002.90.1] 2020-11-29T16:21:44.420 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:44.414+0000 7f42d880f080 0 bucket checkpoint complete 2020-11-29T16:21:44.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:44.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-63', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:44.449 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-63 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:44.496 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:44.497 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:44.773 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:44.767+0000 7f26abaee080 1 bucket sync caught up with source: 2020-11-29T16:21:44.773 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000004.70.5, 00000000002.30.1, 00000000004.53.5, 00000000002.101.1, 00000000004.50.5, 00000000002.104.1, 00000000002.58.1, 00000000004.48.5, 00000000002.57.1, 00000000002.54.1, 00000000002.90.1] 2020-11-29T16:21:44.773 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000004.70.5, 00000000002.30.1, 00000000004.53.5, 00000000002.101.1, 00000000004.50.5, 00000000002.104.1, 00000000002.58.1, 00000000004.48.5, 00000000002.57.1, 00000000002.54.1, 00000000002.90.1] 2020-11-29T16:21:44.773 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:44.767+0000 7f26abaee080 0 bucket checkpoint complete 2020-11-29T16:21:44.805 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-63 zones={test-zone2, test-zone1} 2020-11-29T16:21:44.816 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:44.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:44.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:44.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:44.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:44.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:44.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:44.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:44.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:44.827 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:44.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:44.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:44.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:44.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:44.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:44.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:44.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:44.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:44.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:44.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:45.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:45.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:45.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:45.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:45.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:45.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:45.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:45.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:45.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:45.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:45.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:46.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:46.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:46.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:46.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:46.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:46.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:46.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:46.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:46.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:46.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:47.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:47.118 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-63 zones={test-zone2, test-zone1} 2020-11-29T16:21:47.119 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-63 zones={test-zone3, test-zone1} 2020-11-29T16:21:47.128 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:47.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:47.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:47.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:47.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:47.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:47.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:47.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:47.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:47.136 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:47.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:47.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:47.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:47.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:47.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:47.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:47.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:47.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:47.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:47.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:47.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:47.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:47.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:47.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:47.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:47.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:47.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:47.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:48.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:48.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:48.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:48.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:48.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:48.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:48.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:48.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:48.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:48.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:48.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:49.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:49.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:49.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:49.334 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-63 zones={test-zone3, test-zone1} 2020-11-29T16:21:49.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-63 zones={test-zone3, test-zone2} 2020-11-29T16:21:49.345 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:49.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:49.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:49.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:49.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:49.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:49.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:49.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:49.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:49.354 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:49.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:49.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:49.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:49.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:49.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:49.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:49.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:49.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:49.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:49.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:49.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:49.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:49.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:49.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:49.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:50.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:50.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:50.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:50.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:50.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:50.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:50.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:50.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:50.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:50.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:50.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:51.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:51.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:51.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:51.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:51.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:51.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:51.617 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-63 zones={test-zone3, test-zone2} 2020-11-29T16:21:51.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:51.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:51.618 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:51.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:51.666 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:51.920 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:51.915+0000 7f690cd68080 1 bucket sync caught up with source: 2020-11-29T16:21:51.921 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.84.5, 00000000004.81.1, 00000000006.86.5, 00000000004.44.1, 00000000006.67.5, 00000000004.37.1, 00000000004.42.1, 00000000006.75.5, 00000000004.44.1, 00000000004.52.1, 00000000004.52.1] 2020-11-29T16:21:51.921 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.84.5, 00000000004.81.1, 00000000006.86.5, 00000000004.44.1, 00000000006.67.5, 00000000004.37.1, 00000000004.42.1, 00000000006.75.5, 00000000004.44.1, 00000000004.52.1, 00000000004.52.1] 2020-11-29T16:21:51.921 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:51.915+0000 7f690cd68080 0 bucket checkpoint complete 2020-11-29T16:21:51.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:51.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:51.952 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:51.999 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:52.000 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:52.250 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:52.241+0000 7f5009d70080 1 bucket sync caught up with source: 2020-11-29T16:21:52.251 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000006.84.5, 00000000004.81.1, 00000000006.86.5, 00000000004.44.1, 00000000006.67.5, 00000000004.37.1, 00000000004.42.1, 00000000006.75.5, 00000000004.44.1, 00000000004.52.1, 00000000004.52.1] 2020-11-29T16:21:52.251 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000006.84.5, 00000000004.81.1, 00000000006.86.5, 00000000004.44.1, 00000000006.67.5, 00000000004.37.1, 00000000004.42.1, 00000000006.75.5, 00000000004.44.1, 00000000004.52.1, 00000000004.52.1] 2020-11-29T16:21:52.251 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:52.241+0000 7f5009d70080 0 bucket checkpoint complete 2020-11-29T16:21:52.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:52.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:52.282 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:52.328 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:52.328 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:52.560 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:52.555+0000 7f22a93e4080 1 bucket sync caught up with source: 2020-11-29T16:21:52.561 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000006.84.5, 00000000004.81.1, 00000000006.86.5, 00000000004.44.1, 00000000006.67.5, 00000000004.37.1, 00000000004.42.1, 00000000006.75.5, 00000000004.44.1, 00000000004.52.1, 00000000004.52.1] 2020-11-29T16:21:52.561 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000006.84.5, 00000000004.81.1, 00000000006.86.5, 00000000004.44.1, 00000000006.67.5, 00000000004.37.1, 00000000004.42.1, 00000000006.75.5, 00000000004.44.1, 00000000004.52.1, 00000000004.52.1] 2020-11-29T16:21:52.561 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:52.555+0000 7f22a93e4080 0 bucket checkpoint complete 2020-11-29T16:21:52.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:52.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:52.596 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:52.644 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:52.644 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:52.892 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:52.885+0000 7fc4eb82d080 1 bucket sync caught up with source: 2020-11-29T16:21:52.893 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.86.5, 00000000002.78.1, 00000000004.88.5, 00000000002.56.1, 00000000004.82.5, 00000000002.41.1, 00000000002.37.1, 00000000004.76.5, 00000000002.38.1, 00000000002.47.1, 00000000002.38.1] 2020-11-29T16:21:52.893 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.86.5, 00000000002.78.1, 00000000004.88.5, 00000000002.56.1, 00000000004.82.5, 00000000002.41.1, 00000000002.37.1, 00000000004.76.5, 00000000002.38.1, 00000000002.47.1, 00000000002.38.1] 2020-11-29T16:21:52.893 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:52.885+0000 7fc4eb82d080 0 bucket checkpoint complete 2020-11-29T16:21:52.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:52.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:52.930 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:52.980 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:52.981 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:53.240 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:53.233+0000 7fcb32ecb080 1 bucket sync caught up with source: 2020-11-29T16:21:53.240 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.86.5, 00000000002.78.1, 00000000004.88.5, 00000000002.56.1, 00000000004.82.5, 00000000002.41.1, 00000000002.37.1, 00000000004.76.5, 00000000002.38.1, 00000000002.47.1, 00000000002.38.1] 2020-11-29T16:21:53.240 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.86.5, 00000000002.78.1, 00000000004.88.5, 00000000002.56.1, 00000000004.82.5, 00000000002.41.1, 00000000002.37.1, 00000000004.76.5, 00000000002.38.1, 00000000002.47.1, 00000000002.38.1] 2020-11-29T16:21:53.240 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:53.233+0000 7fcb32ecb080 0 bucket checkpoint complete 2020-11-29T16:21:53.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:53.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:53.272 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:53.320 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:53.321 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:53.579 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:53.569+0000 7f1527dad080 1 bucket sync caught up with source: 2020-11-29T16:21:53.579 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [00000000004.91.1, 00000000002.92.1, 00000000004.93.1, 00000000002.48.1, 00000000004.81.1, 00000000002.43.1, 00000000002.38.1, 00000000004.81.1, 00000000002.40.1, 00000000002.51.1, 00000000002.47.1] 2020-11-29T16:21:53.580 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [00000000004.91.1, 00000000002.92.1, 00000000004.93.1, 00000000002.48.1, 00000000004.81.1, 00000000002.43.1, 00000000002.38.1, 00000000004.81.1, 00000000002.40.1, 00000000002.51.1, 00000000002.47.1] 2020-11-29T16:21:53.580 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:21:53.569+0000 7f1527dad080 0 bucket checkpoint complete 2020-11-29T16:21:53.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:53.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-64', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:21:53.611 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-64 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:21:53.659 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:21:53.659 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:21:53.916 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:53.911+0000 7f46c7763080 1 bucket sync caught up with source: 2020-11-29T16:21:53.917 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000004.91.1, 00000000002.92.1, 00000000004.93.1, 00000000002.48.1, 00000000004.81.1, 00000000002.43.1, 00000000002.38.1, 00000000004.81.1, 00000000002.40.1, 00000000002.51.1, 00000000002.47.1] 2020-11-29T16:21:53.917 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000004.91.1, 00000000002.92.1, 00000000004.93.1, 00000000002.48.1, 00000000004.81.1, 00000000002.43.1, 00000000002.38.1, 00000000004.81.1, 00000000002.40.1, 00000000002.51.1, 00000000002.47.1] 2020-11-29T16:21:53.918 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:21:53.911+0000 7f46c7763080 0 bucket checkpoint complete 2020-11-29T16:21:53.949 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-64 zones={test-zone2, test-zone1} 2020-11-29T16:21:53.961 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:53.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:53.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:53.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:53.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:53.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:53.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:53.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:53.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:53.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:53.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:53.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:53.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:53.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:53.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:53.986 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:53.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:53.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:54.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:54.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:54.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:54.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:54.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:54.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:54.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:54.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:54.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:54.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:54.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:55.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:55.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:55.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:55.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:55.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:55.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:55.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:55.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:55.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:55.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:55.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:56.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:56.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:56.237 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-64 zones={test-zone2, test-zone1} 2020-11-29T16:21:56.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-64 zones={test-zone3, test-zone1} 2020-11-29T16:21:56.246 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:56.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:56.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:56.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:56.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:56.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:56.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:56.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:56.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:56.254 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:56.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:56.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:56.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:56.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:56.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:56.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:56.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:56.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:56.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:56.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:56.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:56.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:56.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:56.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:56.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:56.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:57.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:57.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:57.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:57.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:57.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:57.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:57.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:57.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:57.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:57.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:21:57.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:58.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:58.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:21:58.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:58.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:58.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:21:58.522 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-64 zones={test-zone3, test-zone1} 2020-11-29T16:21:58.522 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-64 zones={test-zone3, test-zone2} 2020-11-29T16:21:58.533 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:21:58.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:58.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:58.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:58.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:58.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:58.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:58.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:58.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:58.541 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:21:58.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-11-29T16:21:58.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-11-29T16:21:58.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-11-29T16:21:58.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-11-29T16:21:58.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-11-29T16:21:58.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-11-29T16:21:58.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-11-29T16:21:58.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-11-29T16:21:58.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:58.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:58.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-11-29T16:21:58.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:58.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:59.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-11-29T16:21:59.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:59.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:59.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-11-29T16:21:59.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:59.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:59.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-11-29T16:21:59.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:59.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:59.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-11-29T16:21:59.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:22:00.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:22:00.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-11-29T16:22:00.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:22:00.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:22:00.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-11-29T16:22:00.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:22:00.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:22:00.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-11-29T16:22:00.793 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-64 zones={test-zone3, test-zone2} 2020-11-29T16:22:00.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:00.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:00.795 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:00.837 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:00.837 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:01.120 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:22:01.121 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:01.121 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-29T16:22:01.121 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:18.806631Z" 2020-11-29T16:22:01.121 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:01.122 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:01.122 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-29T16:22:01.122 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:07.880370Z" 2020-11-29T16:22:01.122 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:01.123 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:01.123 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-29T16:22:01.123 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:20.307494Z" 2020-11-29T16:22:01.123 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:01.124 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:01.124 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000012204", 2020-11-29T16:22:01.124 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:37.207855Z" 2020-11-29T16:22:01.124 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:01.125 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:22:01.126 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:22:01.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:01.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:01.127 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:01.170 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:01.171 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:01.452 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:22:01.452 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:22:01.453 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:22:01.453 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:22:01.453 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:22:01.453 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 6162550666273865027 2020-11-29T16:22:01.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:01.454 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:22:01.454 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:01.454 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:22:01.455 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:01.455 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:01.455 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-29T16:22:01.455 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:01.455 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:01.456 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:01.456 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:18.806631Z" 2020-11-29T16:22:01.456 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:01.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:01.457 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:01.457 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:22:01.457 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:01.457 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:01.457 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-29T16:22:01.457 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:01.458 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:01.458 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:01.458 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:07.880370Z" 2020-11-29T16:22:01.458 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:01.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:01.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:01.459 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:22:01.459 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:01.459 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:01.460 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-29T16:22:01.460 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:01.460 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:01.460 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:01.460 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:20.307494Z" 2020-11-29T16:22:01.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:01.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:01.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:01.461 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:22:01.461 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:01.462 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:01.462 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000012204", 2020-11-29T16:22:01.462 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:01.462 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:01.462 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:01.463 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:37.207855Z" 2020-11-29T16:22:01.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:01.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:01.463 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:22:01.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:01.464 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:22:01.464 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:22:01.464 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:22:01.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:01.464 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:22:01.465 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-11-29T16:22:01.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:01.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:01.467 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:01.506 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:01.507 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:01.792 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:22:01.792 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:01.792 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-29T16:22:01.793 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:18.806631Z" 2020-11-29T16:22:01.793 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:01.793 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:01.793 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-29T16:22:01.794 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:07.880370Z" 2020-11-29T16:22:01.794 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:01.794 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:01.794 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-29T16:22:01.795 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:20.307494Z" 2020-11-29T16:22:01.795 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:01.795 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:01.795 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000012204", 2020-11-29T16:22:01.795 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:37.207855Z" 2020-11-29T16:22:01.796 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:01.796 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:22:01.797 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:22:01.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:01.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:01.799 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:01.881 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:01.882 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:02.169 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:22:02.169 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:22:02.169 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:22:02.170 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:22:02.170 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:22:02.170 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 5268709521422156746 2020-11-29T16:22:02.170 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:02.171 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:22:02.171 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:02.171 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:22:02.171 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:02.171 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:02.172 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-29T16:22:02.172 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:02.172 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:02.172 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:02.173 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:18.806631Z" 2020-11-29T16:22:02.173 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:02.173 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:02.173 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:02.173 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:22:02.174 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:02.174 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:02.174 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-29T16:22:02.174 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:02.174 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:02.175 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:02.175 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:07.880370Z" 2020-11-29T16:22:02.175 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:02.175 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:02.175 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:02.176 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:22:02.176 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:02.176 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:02.176 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-29T16:22:02.176 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:02.177 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:02.177 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:02.177 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:20.307494Z" 2020-11-29T16:22:02.177 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:02.177 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:02.178 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:02.178 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:22:02.178 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:02.178 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:02.179 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000012204", 2020-11-29T16:22:02.179 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:02.179 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:02.179 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:02.179 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:37.207855Z" 2020-11-29T16:22:02.180 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:02.180 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:02.180 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:22:02.180 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:02.180 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:22:02.181 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:22:02.181 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:22:02.181 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:02.181 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:22:02.182 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-11-29T16:22:02.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:02.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:02.184 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:02.262 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:02.262 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:02.548 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:22:02.549 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:02.549 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-29T16:22:02.549 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:18.806631Z" 2020-11-29T16:22:02.550 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:02.550 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:02.550 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-29T16:22:02.550 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:07.880370Z" 2020-11-29T16:22:02.550 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:02.551 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:02.551 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-29T16:22:02.551 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:20.307494Z" 2020-11-29T16:22:02.551 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:02.552 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:02.552 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000012204", 2020-11-29T16:22:02.552 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:37.207855Z" 2020-11-29T16:22:02.552 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:02.552 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:22:02.553 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:22:02.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:02.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:02.555 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:02.597 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:02.597 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:02.863 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:22:02.863 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:22:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:22:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:22:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:22:02.864 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 16089441476439148043 2020-11-29T16:22:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:22:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:02.865 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:22:02.866 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:02.866 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:02.866 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018845", 2020-11-29T16:22:02.866 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:02.867 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:02.867 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:02.867 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:18.806631Z" 2020-11-29T16:22:02.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:02.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:02.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:02.868 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:22:02.868 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:02.869 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:02.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000018047", 2020-11-29T16:22:02.869 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:02.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:02.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:02.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:07.880370Z" 2020-11-29T16:22:02.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:02.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:02.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:02.871 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:22:02.871 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:02.871 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:02.871 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000014359", 2020-11-29T16:22:02.872 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:02.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:02.872 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:02.872 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:20.307494Z" 2020-11-29T16:22:02.873 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:02.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:02.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:02.873 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:22:02.873 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:02.874 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:02.874 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000012204", 2020-11-29T16:22:02.874 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:02.875 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:02.875 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:02.875 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:37.207855Z" 2020-11-29T16:22:02.875 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:02.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:02.876 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:22:02.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:02.876 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:22:02.876 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:22:02.877 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:22:02.877 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:02.877 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:22:02.878 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-11-29T16:22:02.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:02.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:02.880 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:02.955 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:02.955 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:03.239 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:22:03.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:03.240 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000009677", 2020-11-29T16:22:03.240 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:21:15.244209Z" 2020-11-29T16:22:03.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:03.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:03.241 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008880", 2020-11-29T16:22:03.242 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:21:15.249039Z" 2020-11-29T16:22:03.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:03.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:03.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005435", 2020-11-29T16:22:03.243 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:21:15.256551Z" 2020-11-29T16:22:03.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:03.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:03.244 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006092", 2020-11-29T16:22:03.244 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:21:15.260860Z" 2020-11-29T16:22:03.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:03.245 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:22:03.246 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:22:03.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:03.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:03.247 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:03.290 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:03.291 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:03.568 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:22:03.569 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:22:03.569 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:22:03.569 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:22:03.569 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:22:03.570 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 201246450793848048 2020-11-29T16:22:03.570 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:03.570 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:22:03.570 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:03.570 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:22:03.571 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:03.571 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:03.571 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000009677", 2020-11-29T16:22:03.571 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:03.571 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:03.572 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:03.572 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:15.244209Z" 2020-11-29T16:22:03.572 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:03.572 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:03.573 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:03.573 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:22:03.573 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:03.573 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:03.573 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008880", 2020-11-29T16:22:03.574 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:03.574 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:03.574 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:03.574 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:15.249039Z" 2020-11-29T16:22:03.575 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:03.575 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:03.575 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:03.575 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:22:03.576 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:03.576 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:03.577 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005435", 2020-11-29T16:22:03.577 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:03.578 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:03.578 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:03.579 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:15.256551Z" 2020-11-29T16:22:03.579 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:03.580 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:03.580 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:03.580 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:22:03.581 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:03.581 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:03.581 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006092", 2020-11-29T16:22:03.581 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:03.581 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:03.582 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:03.582 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:15.260860Z" 2020-11-29T16:22:03.582 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:03.582 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:03.583 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:22:03.583 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:03.583 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:22:03.583 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:22:03.583 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:22:03.584 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:03.584 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:22:03.585 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-11-29T16:22:03.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:03.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:03.586 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:03.631 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:03.632 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:03.898 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-11-29T16:22:03.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:03.899 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000009677", 2020-11-29T16:22:03.899 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:21:15.244209Z" 2020-11-29T16:22:03.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:03.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:03.900 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008880", 2020-11-29T16:22:03.900 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:21:15.249039Z" 2020-11-29T16:22:03.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:03.901 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:03.901 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005435", 2020-11-29T16:22:03.901 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:21:15.256551Z" 2020-11-29T16:22:03.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:03.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:03.902 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006092", 2020-11-29T16:22:03.902 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-11-29T16:21:15.260860Z" 2020-11-29T16:22:03.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:03.903 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-11-29T16:22:03.904 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:22:03.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:03.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:03.905 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:03.949 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:03.949 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:04.222 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:22:04.222 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:22:04.222 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:22:04.223 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:22:04.223 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:22:04.223 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 16876182154419717581 2020-11-29T16:22:04.224 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.224 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:22:04.224 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.224 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:22:04.225 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:04.225 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:04.225 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000009677", 2020-11-29T16:22:04.225 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:04.226 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:04.226 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:04.226 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:15.244209Z" 2020-11-29T16:22:04.227 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.228 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.228 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.228 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:22:04.228 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:04.229 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:04.229 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000008880", 2020-11-29T16:22:04.229 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:04.229 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:04.230 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:04.230 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:15.249039Z" 2020-11-29T16:22:04.230 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.230 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.230 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.231 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:22:04.231 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:04.231 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:04.231 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000005435", 2020-11-29T16:22:04.231 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:04.232 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:04.232 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:04.232 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:15.256551Z" 2020-11-29T16:22:04.232 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.233 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.233 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.233 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:22:04.233 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:04.233 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:04.234 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000006092", 2020-11-29T16:22:04.234 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:04.234 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:04.234 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:04.234 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:15.260860Z" 2020-11-29T16:22:04.235 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.235 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.235 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:22:04.235 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.235 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:22:04.236 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:22:04.236 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:22:04.236 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.236 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:22:04.237 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-11-29T16:22:04.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:04.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:04.239 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:04.318 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:04.319 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:04.602 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:22:04.602 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.603 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000011273", 2020-11-29T16:22:04.603 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:36.829428Z" 2020-11-29T16:22:04.604 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.604 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.604 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000011274", 2020-11-29T16:22:04.604 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:14.814418Z" 2020-11-29T16:22:04.605 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.605 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.605 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007437", 2020-11-29T16:22:04.605 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:36.834565Z" 2020-11-29T16:22:04.606 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.606 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.607 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007023", 2020-11-29T16:22:04.607 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:36.839279Z" 2020-11-29T16:22:04.607 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.608 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:22:04.609 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:22:04.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:04.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:04.611 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:04.692 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:04.693 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:04.966 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:22:04.967 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:22:04.967 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:22:04.967 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:22:04.967 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:22:04.967 INFO:teuthology.orchestra.run.smithi168.stdout: "instance_id": 3114293326551352523 2020-11-29T16:22:04.968 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.968 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:22:04.968 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.969 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:22:04.969 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:04.969 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:04.969 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000011273", 2020-11-29T16:22:04.970 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:04.970 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:04.970 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:04.970 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:36.829428Z" 2020-11-29T16:22:04.971 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.971 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.971 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.971 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:22:04.971 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:04.972 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:04.972 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000011274", 2020-11-29T16:22:04.972 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:04.972 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:04.972 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:04.973 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:14.814418Z" 2020-11-29T16:22:04.973 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.973 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.973 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.974 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:22:04.974 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:04.974 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:04.974 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007437", 2020-11-29T16:22:04.975 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:04.975 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:04.975 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:04.975 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:36.834565Z" 2020-11-29T16:22:04.976 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.977 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.977 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:04.977 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:22:04.977 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:22:04.977 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "incremental-sync", 2020-11-29T16:22:04.978 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007023", 2020-11-29T16:22:04.978 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:22:04.978 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:22:04.978 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:22:04.979 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:21:36.839279Z" 2020-11-29T16:22:04.979 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.979 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.979 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:22:04.980 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:04.980 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:22:04.980 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 0, 2020-11-29T16:22:04.980 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 0 2020-11-29T16:22:04.980 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:04.981 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:22:04.982 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-11-29T16:22:04.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:04.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:04.984 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:05.062 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:05.063 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:05.361 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:22:05.362 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:05.362 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000011273", 2020-11-29T16:22:05.362 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:36.829428Z" 2020-11-29T16:22:05.362 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:05.363 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:05.363 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000011274", 2020-11-29T16:22:05.363 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:14.814418Z" 2020-11-29T16:22:05.363 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:05.363 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:05.364 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007437", 2020-11-29T16:22:05.364 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:36.834565Z" 2020-11-29T16:22:05.364 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:22:05.364 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:22:05.365 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "00000000000000000000:00000000000000007023", 2020-11-29T16:22:05.365 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:21:36.839279Z" 2020-11-29T16:22:05.365 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:22:05.365 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:22:05.366 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:22:05.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:05.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:22:05.368 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:22:05.412 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:05.412 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:05.710 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:22:05.711 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:22:05.711 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:22:05.711 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:22:05.711 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:22:05.711 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 2748347558345283993 2020-11-29T16:22:05.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:05.712 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:22:05.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:05.712 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:22:05.713 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:05.713 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:05.713 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000011273", 2020-11-29T16:22:05.713 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:05.713 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:05.714 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:05.714 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:36.829428Z" 2020-11-29T16:22:05.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:05.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:05.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:05.715 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:22:05.715 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:05.715 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000011274", 2020-11-29T16:22:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:14.814418Z" 2020-11-29T16:22:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:22:05.718 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:05.718 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:05.718 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000007437", 2020-11-29T16:22:05.718 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:05.718 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:36.834565Z" 2020-11-29T16:22:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:22:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:22:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:22:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-11-29T16:22:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000007023", 2020-11-29T16:22:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:22:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:22:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:22:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:21:36.839279Z" 2020-11-29T16:22:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:22:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:22:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:22:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-11-29T16:22:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-11-29T16:22:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:22:05.723 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:22:05.724 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-11-29T16:22:05.725 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2020-11-29T16:22:05.725 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-65 2020-11-29T16:22:05.773 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=kcmyep-66 2020-11-29T16:22:05.858 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=kcmyep-67 2020-11-29T16:22:08.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:08.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:08.140 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:22:08.188 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:08.188 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:08.464 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:08.459+0000 7f66cb55c080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:08.464 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.86.5, , , , , ] 2020-11-29T16:22:08.465 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:13.468 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:13.463+0000 7f66cb55c080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:13.469 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.86.5, , , , , ] 2020-11-29T16:22:13.469 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:18.472 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:18.467+0000 7f66cb55c080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:18.472 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.86.5, , , , , ] 2020-11-29T16:22:18.472 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:23.476 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:23.471+0000 7f66cb55c080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:23.476 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.86.5, , , , , ] 2020-11-29T16:22:23.476 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:28.480 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:28.474+0000 7f66cb55c080 1 bucket sync caught up with source: 2020-11-29T16:22:28.481 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:28.481 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:28.481 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:28.474+0000 7f66cb55c080 0 bucket checkpoint complete 2020-11-29T16:22:28.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:28.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:28.516 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:22:28.565 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:28.566 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:28.840 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:22:28.837+0000 7f07e90e0080 1 bucket sync caught up with source: 2020-11-29T16:22:28.840 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:28.840 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:28.841 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:22:28.837+0000 7f07e90e0080 0 bucket checkpoint complete 2020-11-29T16:22:28.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:28.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:28.872 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:22:28.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:28.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:29.163 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:29.154+0000 7fefffe23080 1 bucket sync caught up with source: 2020-11-29T16:22:29.163 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:29.164 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.90.5, , , , , ] 2020-11-29T16:22:29.164 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:29.154+0000 7fefffe23080 0 bucket checkpoint complete 2020-11-29T16:22:29.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:29.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:29.289 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:22:29.342 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:29.343 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:29.615 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:22:29.609+0000 7f8c63dc1080 1 bucket sync caught up with source: 2020-11-29T16:22:29.615 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000001.81.5, , , , , ] 2020-11-29T16:22:29.616 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000001.81.5, , , , , ] 2020-11-29T16:22:29.616 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:22:29.609+0000 7f8c63dc1080 0 bucket checkpoint complete 2020-11-29T16:22:29.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:29.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:29.654 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-65 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:22:29.703 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:29.703 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:29.982 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:22:29.977+0000 7f5c9edb8080 1 bucket sync caught up with source: 2020-11-29T16:22:29.982 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000001.81.5, , , , , ] 2020-11-29T16:22:29.982 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000001.81.5, , , , , ] 2020-11-29T16:22:29.983 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:22:29.977+0000 7f5c9edb8080 0 bucket checkpoint complete 2020-11-29T16:22:30.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:30.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:30.016 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:22:30.063 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:30.063 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:30.319 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:22:30.317+0000 7f92c59e7080 1 bucket sync caught up with source: 2020-11-29T16:22:30.320 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:22:30.320 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:22:30.320 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:22:30.317+0000 7f92c59e7080 0 bucket checkpoint complete 2020-11-29T16:22:30.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:30.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-65', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:22:30.351 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-65 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:22:30.398 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:22:30.398 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:22:30.672 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:30.666+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:30.672 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:22:30.673 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:22:35.676 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:35.670+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:35.676 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:22:35.676 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:22:40.680 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:40.674+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:40.680 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:22:40.680 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:22:45.683 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:45.678+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:45.684 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:22:45.684 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:22:50.687 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:50.678+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:50.687 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:22:50.687 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:22:55.690 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:22:55.682+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:22:55.691 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:22:55.691 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:00.694 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:00.686+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:00.695 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:00.695 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:05.698 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:05.690+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:05.698 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:05.699 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:10.702 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:10.693+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:10.702 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:10.703 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:15.705 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:15.697+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:15.706 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:15.706 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:20.709 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:20.701+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:20.709 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:20.710 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:25.713 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:25.705+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:25.713 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:25.713 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:30.716 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:30.709+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:30.717 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:30.717 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:35.720 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:35.713+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:35.721 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:35.721 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:40.724 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:40.717+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:40.724 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:40.724 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:45.728 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:45.721+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:45.728 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:45.728 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:50.733 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:50.725+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:50.733 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:50.733 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:23:55.737 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:23:55.732+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:23:55.737 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:23:55.737 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:00.741 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:00.736+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:24:00.741 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:24:00.741 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:05.744 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:05.740+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:24:05.745 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:24:05.745 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:10.748 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:10.744+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:24:10.749 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:24:10.749 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:15.753 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:15.748+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:24:15.753 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:24:15.754 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:20.757 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:20.752+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:24:20.757 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:24:20.757 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:25.761 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:25.756+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:24:25.761 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:24:25.761 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:30.764 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:30.760+0000 7fbbd1c2a080 1 waiting for incremental sync to catch up: 2020-11-29T16:24:30.765 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-11-29T16:24:30.765 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:35.768 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:35.763+0000 7fbbd1c2a080 1 bucket sync caught up with source: 2020-11-29T16:24:35.769 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:35.769 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000001.89.5, , , , , ] 2020-11-29T16:24:35.769 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:35.763+0000 7fbbd1c2a080 0 bucket checkpoint complete 2020-11-29T16:24:35.801 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-65 zones={test-zone2, test-zone1} 2020-11-29T16:24:35.815 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:35.862 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-29T16:24:35.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:35.871 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-29T16:24:35.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:36.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:36.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:36.685 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-65 zones={test-zone2, test-zone1} 2020-11-29T16:24:36.685 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-65 zones={test-zone3, test-zone1} 2020-11-29T16:24:36.694 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:36.746 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-29T16:24:36.747 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:36.753 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-29T16:24:36.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:37.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:37.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:37.642 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-65 zones={test-zone3, test-zone1} 2020-11-29T16:24:37.643 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-65 zones={test-zone3, test-zone2} 2020-11-29T16:24:37.654 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:37.702 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-29T16:24:37.703 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:37.711 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-11-29T16:24:37.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:38.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:38.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-11-29T16:24:38.678 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-65 zones={test-zone3, test-zone2} 2020-11-29T16:24:38.679 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2020-11-29T16:24:38.680 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-68 2020-11-29T16:24:38.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:38.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:38.820 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:38.859 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:38.860 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:38.920 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T16:24:38.915+0000 7f9a87fef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:38.920 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T16:24:38.915+0000 7f9a86fed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:38.921 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:24:38.915+0000 7fec017ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:38.922 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:24:38.915+0000 7fec087f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:38.922 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T16:24:38.915+0000 7fc9baffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:38.922 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T16:24:38.915+0000 7fc9afff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:39.090 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:24:39.121 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:24:39.121 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:39.121 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:39.122 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:39.122 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:39.122 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:39.122 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667078.720701_460.1", 2020-11-29T16:24:39.122 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:24:38.720701Z" 2020-11-29T16:24:39.123 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:39.123 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:39.123 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:39.123 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:39.123 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:39.124 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:39.124 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:39.124 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:39.124 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:39.124 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:24:39.125 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:24:39.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:39.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:39.127 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:39.170 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:39.171 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:39.430 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:24:39.431 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:24:39.431 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:24:39.432 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:24:39.432 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:24:39.432 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:39.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:39.434 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:24:39.434 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:39.434 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:24:39.435 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:39.435 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:39.436 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:39.436 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:39.436 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:24:39.437 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:39.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:39.438 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:39.438 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:39.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:39.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:39.439 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:24:39.439 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:39.440 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:39.440 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667078.720701_460.1", 2020-11-29T16:24:39.441 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:39.441 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:39.441 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:39.442 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:38.720701Z", 2020-11-29T16:24:39.442 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:39.443 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:39.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:39.444 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:39.444 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:24:39.444 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:39.445 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:39.445 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:39.446 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:39.446 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:39.446 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:39.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:39.447 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:39.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:39.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:39.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:24:39.449 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:39.450 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:39.451 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:39.451 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:39.452 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:39.452 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:39.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:39.453 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:39.453 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:24:39.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:39.454 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:24:39.454 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:24:39.455 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:24:39.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:39.455 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:24:39.457 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:24:39.458 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:24:39.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:39.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:39.460 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:39.502 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:39.503 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:39.782 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:24:39.783 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:24:39.783 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:24:39.783 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:24:39.783 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:24:39.784 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:39.784 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:24:39.784 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:39.784 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:24:39.784 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:39.785 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:24:39.785 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:39.785 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:39.785 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:39.786 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:39.786 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:24:39.786 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:39.786 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:39.786 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:39.787 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:39.787 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:39.787 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:39.787 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:24:39.787 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:39.788 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:39.788 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667078.720701_460.1", 2020-11-29T16:24:39.788 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:39.788 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:39.789 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:39.789 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:24:38.720701Z", 2020-11-29T16:24:39.789 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:24:39.789 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:39.789 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:39.790 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:39.790 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:24:39.790 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:39.790 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:39.791 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:39.791 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:39.791 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:39.791 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:39.791 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:24:39.792 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:39.792 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:39.792 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:39.792 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:39.792 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:24:39.793 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:39.793 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:39.793 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:39.793 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:39.794 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:39.794 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:39.794 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:24:39.794 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:39.794 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:39.795 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:39.795 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:24:39.795 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:39.795 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:24:39.795 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:24:39.796 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:24:39.796 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:39.796 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:24:39.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:24:39.798 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:24:39.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:39.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:39.799 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:39.840 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:39.841 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:40.121 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:24:40.122 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:24:40.122 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:24:40.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:24:40.122 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:24:40.123 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:40.123 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:40.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:40.123 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:24:40.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:40.124 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:24:40.124 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:40.125 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:40.125 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:40.125 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:40.125 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:24:40.126 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:40.126 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:40.126 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:40.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:40.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:40.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:40.127 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:24:40.127 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:40.127 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:40.128 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667078.720701_460.1", 2020-11-29T16:24:40.128 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:40.128 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:40.128 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:40.129 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:38.720701Z", 2020-11-29T16:24:40.129 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:40.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:40.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:40.129 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:40.130 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:24:40.130 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:40.130 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:40.130 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:40.131 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:40.131 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:40.131 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:40.131 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:40.131 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:40.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:40.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:40.132 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:40.132 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:24:40.133 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:40.133 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:40.133 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:40.133 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:40.133 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:40.134 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:40.134 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:40.134 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:40.134 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:40.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:40.135 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:24:40.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:40.135 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:24:40.135 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:24:40.136 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:24:40.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:40.136 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:24:40.137 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:24:40.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:40.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-68', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:40.139 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-68 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:40.226 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:40.226 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:40.493 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:40.487+0000 7f3f33c42080 1 bucket sync caught up with source: 2020-11-29T16:24:40.494 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.45.5, , , , 00000000001.62.5, , , , , ] 2020-11-29T16:24:40.494 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.45.5, , , , 00000000001.62.5, , , , , ] 2020-11-29T16:24:40.494 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:40.487+0000 7f3f33c42080 0 bucket checkpoint complete 2020-11-29T16:24:40.630 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2020-11-29T16:24:40.631 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-69 2020-11-29T16:24:40.718 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:24:40.711+0000 7fec127fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:40.719 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T16:24:40.711+0000 7f9a887f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:40.719 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T16:24:40.711+0000 7fc9adff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:40.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:40.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:40.872 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:40.912 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:40.912 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:41.165 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:24:41.193 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:24:41.193 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:41.194 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:41.194 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:41.194 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:41.195 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:41.195 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667080.668359_482.1", 2020-11-29T16:24:41.196 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:24:40.668359Z" 2020-11-29T16:24:41.196 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:41.197 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:41.197 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:41.197 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:41.198 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:41.198 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:41.199 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:41.199 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:41.199 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:41.200 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:24:41.201 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:24:41.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:41.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:41.204 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:41.246 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:41.247 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:41.503 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:24:41.503 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:24:41.503 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:24:41.503 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:24:41.504 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:24:41.504 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:41.504 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:41.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:24:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:24:41.505 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:41.506 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:41.506 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:41.506 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:41.506 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:24:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:41.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:24:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:41.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667080.668359_482.1", 2020-11-29T16:24:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:41.509 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:40.668359Z", 2020-11-29T16:24:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:41.510 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:24:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:41.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:41.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:24:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:41.513 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:41.514 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:41.514 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:41.514 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:41.514 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:41.514 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:41.515 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:41.515 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:41.515 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:24:41.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:41.516 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:24:41.516 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:24:41.516 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:24:41.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:41.517 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:24:41.518 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:24:41.518 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:24:41.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:41.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:41.520 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:41.561 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:41.562 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:41.838 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:24:41.838 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:24:41.838 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:24:41.839 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:24:41.839 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:24:41.839 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:41.839 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:24:41.840 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:41.840 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:24:41.840 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:41.840 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:24:41.840 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:41.841 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:41.841 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:41.841 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:41.842 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:24:41.842 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:41.842 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:41.842 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:41.842 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:41.843 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:41.843 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:41.843 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:24:41.843 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:41.843 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:41.844 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667080.668359_482.1", 2020-11-29T16:24:41.844 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:41.844 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:41.844 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:41.845 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:24:40.668359Z", 2020-11-29T16:24:41.845 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:24:41.845 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:41.845 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:41.845 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:41.846 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:24:41.846 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:41.846 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:41.846 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:41.846 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:41.847 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:41.848 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:41.848 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:24:41.848 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:41.848 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:41.849 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:41.849 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:41.849 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:24:41.849 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:41.849 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:41.850 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:41.850 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:41.850 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:41.850 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:41.850 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:24:41.851 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:41.851 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:41.851 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:41.851 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:24:41.851 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:41.852 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:24:41.852 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:24:41.852 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:24:41.852 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:41.853 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:24:41.854 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:24:41.854 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:24:41.855 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:41.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:41.856 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:41.896 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:41.896 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:42.260 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:24:42.261 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:24:42.261 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:24:42.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:24:42.261 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:24:42.262 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:42.262 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:42.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:42.262 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:24:42.263 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:42.263 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:24:42.263 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:42.263 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:42.263 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:42.264 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:42.264 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:24:42.264 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:42.264 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:42.264 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:42.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:42.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:42.265 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:42.265 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:24:42.265 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:42.266 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:42.266 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667080.668359_482.1", 2020-11-29T16:24:42.266 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:42.266 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:42.267 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:42.267 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:40.668359Z", 2020-11-29T16:24:42.267 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:42.267 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:42.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:42.268 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:42.268 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:24:42.268 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:42.268 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:42.269 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:42.269 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:42.269 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:42.269 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:42.269 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:42.270 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:42.270 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:42.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:42.270 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:42.271 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:24:42.271 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:42.271 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:42.271 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:42.272 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:42.272 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:42.272 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:42.272 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:42.272 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:42.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:42.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:42.273 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:24:42.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:42.273 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:24:42.274 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:24:42.274 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:24:42.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:42.274 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:24:42.275 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:24:42.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:42.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:42.277 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:42.367 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:42.368 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:42.641 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:42.635+0000 7f5ea8753080 1 bucket sync caught up with source: 2020-11-29T16:24:42.642 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.120.5, , , , 00000000001.55.5, , , 00000000001.59.5, , 00000000001.48.5] 2020-11-29T16:24:42.642 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.120.5, , , , 00000000001.55.5, , , 00000000001.59.5, , 00000000001.48.5] 2020-11-29T16:24:42.642 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:42.635+0000 7f5ea8753080 0 bucket checkpoint complete 2020-11-29T16:24:42.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:42.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:42.675 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:42.723 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:42.723 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:42.981 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:42.975+0000 7f8ac6084080 1 bucket sync caught up with source: 2020-11-29T16:24:42.982 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.120.5, , , , 00000000001.55.5, , , 00000000001.59.5, , 00000000001.48.5] 2020-11-29T16:24:42.982 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.120.5, , , , 00000000001.55.5, , , 00000000001.59.5, , 00000000001.48.5] 2020-11-29T16:24:42.982 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:42.975+0000 7f8ac6084080 0 bucket checkpoint complete 2020-11-29T16:24:43.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:43.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:43.013 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-69 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:43.059 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:43.060 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:43.328 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:43.323+0000 7f499d26b080 1 bucket sync caught up with source: 2020-11-29T16:24:43.328 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.120.5, , , , 00000000001.55.5, , , 00000000001.59.5, , 00000000001.48.5] 2020-11-29T16:24:43.328 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.120.5, , , , 00000000001.55.5, , , 00000000001.59.5, , 00000000001.48.5] 2020-11-29T16:24:43.329 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:43.323+0000 7f499d26b080 0 bucket checkpoint complete 2020-11-29T16:24:43.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:43.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:43.439 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:43.487 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:43.488 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:43.758 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:43.751+0000 7f4a747a3080 1 bucket sync caught up with source: 2020-11-29T16:24:43.758 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.145.5, , , , 00000000001.39.5, , , 00000000001.66.5, , 00000000001.55.5] 2020-11-29T16:24:43.759 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.145.5, , , , 00000000001.39.5, , , 00000000001.66.5, , 00000000001.55.5] 2020-11-29T16:24:43.759 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:43.751+0000 7f4a747a3080 0 bucket checkpoint complete 2020-11-29T16:24:43.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:43.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:43.794 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-69 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:43.841 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:43.841 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:44.109 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:44.103+0000 7f6a7ca66080 1 bucket sync caught up with source: 2020-11-29T16:24:44.109 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.145.5, , , , 00000000001.39.5, , , 00000000001.66.5, , 00000000001.55.5] 2020-11-29T16:24:44.109 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.145.5, , , , 00000000001.39.5, , , 00000000001.66.5, , 00000000001.55.5] 2020-11-29T16:24:44.109 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:44.103+0000 7f6a7ca66080 0 bucket checkpoint complete 2020-11-29T16:24:44.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:44.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:44.143 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:44.187 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:44.188 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:44.438 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:44.431+0000 7fb2688c2080 1 bucket sync caught up with source: 2020-11-29T16:24:44.438 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.149.5, , , , 00000000001.39.5, , , 00000000001.65.5, , 00000000001.54.5] 2020-11-29T16:24:44.438 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.149.5, , , , 00000000001.39.5, , , 00000000001.65.5, , 00000000001.54.5] 2020-11-29T16:24:44.439 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:44.431+0000 7fb2688c2080 0 bucket checkpoint complete 2020-11-29T16:24:44.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:44.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-69', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:44.469 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-69 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:44.518 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:44.518 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:44.766 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:44.759+0000 7f43d481e080 1 bucket sync caught up with source: 2020-11-29T16:24:44.766 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.149.5, , , , 00000000001.39.5, , , 00000000001.65.5, , 00000000001.54.5] 2020-11-29T16:24:44.766 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.149.5, , , , 00000000001.39.5, , , 00000000001.65.5, , 00000000001.54.5] 2020-11-29T16:24:44.767 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:44.759+0000 7f43d481e080 0 bucket checkpoint complete 2020-11-29T16:24:44.799 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-69 zones={test-zone2, test-zone1} 2020-11-29T16:24:44.812 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:44.858 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:44.859 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:44.859 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:44.860 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:44.860 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:44.868 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:44.869 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:44.869 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:44.870 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:44.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:44.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:45.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:45.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:45.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:45.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:45.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:45.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:45.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:45.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:45.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:45.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:46.011 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-69 zones={test-zone2, test-zone1} 2020-11-29T16:24:46.012 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-69 zones={test-zone3, test-zone1} 2020-11-29T16:24:46.023 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:46.071 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:46.072 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:46.072 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:46.073 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:46.074 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:46.083 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:46.083 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:46.084 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:46.084 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:46.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:46.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:46.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:46.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:46.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:46.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:46.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:46.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:46.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:46.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:47.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:47.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:47.234 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-69 zones={test-zone3, test-zone1} 2020-11-29T16:24:47.234 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-69 zones={test-zone3, test-zone2} 2020-11-29T16:24:47.246 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:47.251 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:47.252 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:47.252 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:47.253 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:47.253 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:47.259 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:47.260 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:47.261 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:47.261 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:47.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:47.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:47.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:47.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:47.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:47.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:47.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:47.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:48.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:48.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:48.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:48.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:48.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-69 zones={test-zone3, test-zone2} 2020-11-29T16:24:48.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:24:48.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:24:48.408 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:24:48.447 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:48.447 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:48.723 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=kcmyep-70 2020-11-29T16:24:48.926 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T16:24:48.919+0000 7f9a8ffff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.926 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T16:24:48.919+0000 7f9a8bff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.926 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T16:24:48.919+0000 7f9a88ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.927 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T16:24:48.919+0000 7fc9b97fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.927 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T16:24:48.919+0000 7fc9c1ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.927 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T16:24:48.919+0000 7fc9b0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.928 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:24:48.923+0000 7fec13fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.928 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:24:48.923+0000 7fec10ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.929 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:24:48.923+0000 7fec00fe9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:24:48.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:48.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:48.971 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:49.013 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:49.014 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:49.300 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:24:49.327 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:24:49.328 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:49.328 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:49.328 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:49.328 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:49.329 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:49.329 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:24:49.329 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:24:48.760284Z" 2020-11-29T16:24:49.329 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:49.329 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:49.330 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:49.330 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:49.330 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:49.330 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:49.330 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:49.331 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:49.331 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:49.331 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:24:49.332 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:24:49.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:49.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:49.334 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:49.377 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:49.378 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:49.642 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:24:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:24:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:24:49.642 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:24:49.643 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:24:49.643 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:49.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:49.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:49.644 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:24:49.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:49.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:24:49.644 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:49.644 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:49.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:49.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:49.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:24:49.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:49.645 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:49.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:49.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:49.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:49.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:49.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:24:49.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:49.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:49.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:24:49.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:49.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:49.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:49.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:48.760284Z", 2020-11-29T16:24:49.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:49.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:49.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:49.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:49.649 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:24:49.649 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:49.649 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:49.650 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:49.651 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:49.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:49.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:49.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:49.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:24:49.652 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:49.652 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:49.652 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:49.652 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:49.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:49.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:49.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:49.653 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:49.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:49.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:49.654 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:24:49.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:49.654 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:24:49.654 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:24:49.655 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:24:49.655 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:49.655 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:24:49.656 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:24:49.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:24:49.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:49.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:49.658 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:49.696 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:49.697 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:49.984 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:24:49.984 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:24:49.984 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:24:49.985 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:24:49.985 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:24:49.985 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:49.985 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:24:49.986 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:49.986 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:24:49.986 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:49.986 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:24:49.987 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:49.987 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:49.987 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:49.987 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:49.988 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:24:49.988 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:49.989 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:49.989 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:49.990 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:49.990 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:49.990 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:49.991 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:24:49.991 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:49.991 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:49.992 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:24:49.992 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:49.993 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:49.993 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:49.993 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:24:48.760284Z", 2020-11-29T16:24:49.994 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:24:49.994 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:49.994 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:49.995 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:49.995 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:24:49.995 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:49.996 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:49.996 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:49.996 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:49.997 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:49.997 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:49.997 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:24:49.997 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:49.998 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:49.998 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:49.998 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:49.998 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:24:49.999 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:24:49.999 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:24:49.999 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:49.999 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:24:50.000 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:24:50.000 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:24:50.000 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:24:50.000 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:24:50.001 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:50.001 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:50.001 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:24:50.001 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:50.001 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:24:50.002 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:24:50.002 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:24:50.002 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:50.002 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:24:50.003 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:24:50.004 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:24:50.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:50.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:50.006 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:50.047 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:50.048 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:50.318 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:24:50.318 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:24:50.319 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:24:50.319 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:24:50.319 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:24:50.319 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:24:50.320 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:50.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:50.320 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:24:50.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:50.321 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:24:50.321 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:50.321 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:50.321 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:24:50.322 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:50.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:24:50.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:50.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:24:50.323 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:50.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:50.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:50.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:50.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:24:50.324 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:50.324 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:50.324 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:24:50.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:50.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:50.325 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:50.325 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:48.760284Z", 2020-11-29T16:24:50.325 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:24:50.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:50.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:50.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:50.326 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:24:50.327 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:50.327 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:50.327 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:50.327 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:50.327 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:50.328 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:50.328 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:50.328 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:50.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:50.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:50.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:24:50.329 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:24:50.329 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:24:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:24:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:24:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:24:50.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:24:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:24:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:24:50.331 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:24:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:24:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:24:50.332 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:24:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:24:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:24:50.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:24:50.333 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:24:50.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:24:50.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:50.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:50.336 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:50.421 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:50.422 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:50.684 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:50.679+0000 7fd25ca68080 1 bucket sync caught up with source: 2020-11-29T16:24:50.684 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.59.5, , , , 00000000001.62.5, , , 00000000001.69.5, , 00000000001.58.5] 2020-11-29T16:24:50.684 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.62.5, , , 00000000001.69.5, , 00000000001.58.5] 2020-11-29T16:24:50.685 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:50.679+0000 7fd25ca68080 0 bucket checkpoint complete 2020-11-29T16:24:50.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:50.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:50.719 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:50.767 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:50.768 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:51.043 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:51.035+0000 7fccd2b30080 1 bucket sync caught up with source: 2020-11-29T16:24:51.043 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.59.5, , , , 00000000001.62.5, , , 00000000001.69.5, , 00000000001.58.5] 2020-11-29T16:24:51.044 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.62.5, , , 00000000001.69.5, , 00000000001.58.5] 2020-11-29T16:24:51.044 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:51.035+0000 7fccd2b30080 0 bucket checkpoint complete 2020-11-29T16:24:51.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:51.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:51.073 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-70 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:51.120 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:51.121 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:51.361 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:51.355+0000 7fc845a3f080 1 bucket sync caught up with source: 2020-11-29T16:24:51.361 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.59.5, , , , 00000000001.62.5, , , 00000000001.69.5, , 00000000001.58.5] 2020-11-29T16:24:51.362 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.59.5, , , , 00000000001.62.5, , , 00000000001.69.5, , 00000000001.58.5] 2020-11-29T16:24:51.362 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:51.355+0000 7fc845a3f080 0 bucket checkpoint complete 2020-11-29T16:24:51.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:51.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:51.491 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:51.538 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:51.538 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:51.805 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:51.798+0000 7f4ff5fbd080 1 bucket sync caught up with source: 2020-11-29T16:24:51.806 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.62.5, , , , 00000000001.72.5, , , 00000000001.67.5, , 00000000001.45.5] 2020-11-29T16:24:51.806 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.62.5, , , , 00000000001.72.5, , , 00000000001.67.5, , 00000000001.45.5] 2020-11-29T16:24:51.806 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:51.798+0000 7f4ff5fbd080 0 bucket checkpoint complete 2020-11-29T16:24:51.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:51.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:51.843 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-70 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:51.890 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:51.891 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:52.166 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:52.158+0000 7fc568bd5080 1 bucket sync caught up with source: 2020-11-29T16:24:52.166 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.62.5, , , , 00000000001.72.5, , , 00000000001.67.5, , 00000000001.45.5] 2020-11-29T16:24:52.166 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.62.5, , , , 00000000001.72.5, , , 00000000001.67.5, , 00000000001.45.5] 2020-11-29T16:24:52.166 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:52.158+0000 7fc568bd5080 0 bucket checkpoint complete 2020-11-29T16:24:52.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:52.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:52.203 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket kcmyep-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:52.250 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:52.251 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:52.517 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:52.510+0000 7f76a7b1a080 1 bucket sync caught up with source: 2020-11-29T16:24:52.518 INFO:teuthology.orchestra.run.smithi168.stderr: local status: [, 00000000001.60.5, , , , 00000000001.71.5, , , 00000000001.66.5, , 00000000001.45.5] 2020-11-29T16:24:52.518 INFO:teuthology.orchestra.run.smithi168.stderr: remote markers: [, 00000000001.60.5, , , , 00000000001.71.5, , , 00000000001.66.5, , 00000000001.45.5] 2020-11-29T16:24:52.518 INFO:teuthology.orchestra.run.smithi168.stderr:2020-11-29T16:24:52.510+0000 7f76a7b1a080 0 bucket checkpoint complete 2020-11-29T16:24:52.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:52.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-70', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:24:52.548 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-70 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:24:52.598 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:52.598 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:52.859 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:52.851+0000 7f58c11a0080 1 bucket sync caught up with source: 2020-11-29T16:24:52.859 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.60.5, , , , 00000000001.71.5, , , 00000000001.66.5, , 00000000001.45.5] 2020-11-29T16:24:52.859 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.60.5, , , , 00000000001.71.5, , , 00000000001.66.5, , 00000000001.45.5] 2020-11-29T16:24:52.860 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:24:52.851+0000 7f58c11a0080 0 bucket checkpoint complete 2020-11-29T16:24:52.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-70 zones={test-zone2, test-zone1} 2020-11-29T16:24:52.904 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:52.913 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:52.914 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:52.915 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:52.916 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:52.916 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:52.966 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:52.967 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:52.968 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:52.968 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:53.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:53.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:53.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:53.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:53.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:53.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:53.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:53.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:53.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:53.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:53.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:54.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:54.162 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-70 zones={test-zone2, test-zone1} 2020-11-29T16:24:54.163 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-70 zones={test-zone3, test-zone1} 2020-11-29T16:24:54.172 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:54.178 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:54.179 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:54.180 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:54.181 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:54.181 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:54.187 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:54.189 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:54.189 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:54.189 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:54.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:54.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:54.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:54.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:54.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:54.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:54.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:54.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:54.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:55.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:55.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:55.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:55.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-70 zones={test-zone3, test-zone1} 2020-11-29T16:24:55.303 INFO:tasks.rgw_multi.zone_rados:comparing bucket=kcmyep-70 zones={test-zone3, test-zone2} 2020-11-29T16:24:55.314 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-11-29T16:24:55.319 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:55.319 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:55.320 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:55.320 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:55.320 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-11-29T16:24:55.327 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-11-29T16:24:55.327 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-11-29T16:24:55.328 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-11-29T16:24:55.328 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-11-29T16:24:55.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:55.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:55.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-11-29T16:24:55.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:55.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:55.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-11-29T16:24:55.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:55.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:56.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-11-29T16:24:56.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:56.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:56.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-11-29T16:24:56.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=kcmyep-70 zones={test-zone3, test-zone2} 2020-11-29T16:24:56.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:24:56.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:24:56.488 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:24:56.526 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:56.526 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:56.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'kcmyep-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:56.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'kcmyep-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:56.828 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket kcmyep-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:56.906 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:56.906 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:57.189 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T16:24:57.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'kcmyep-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:57.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'kcmyep-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:57.192 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket kcmyep-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:57.268 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:57.268 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:57.542 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:24:57.542 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:57.543 INFO:teuthology.orchestra.run.smithi168.stdout: "op_id": "1#00000000001.120.5", 2020-11-29T16:24:57.543 INFO:teuthology.orchestra.run.smithi168.stdout: "op_tag": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5482.5234", 2020-11-29T16:24:57.543 INFO:teuthology.orchestra.run.smithi168.stdout: "op": "write", 2020-11-29T16:24:57.544 INFO:teuthology.orchestra.run.smithi168.stdout: "object": "a", 2020-11-29T16:24:57.544 INFO:teuthology.orchestra.run.smithi168.stdout: "instance": "", 2020-11-29T16:24:57.544 INFO:teuthology.orchestra.run.smithi168.stdout: "state": "complete", 2020-11-29T16:24:57.544 INFO:teuthology.orchestra.run.smithi168.stdout: "index_ver": 1, 2020-11-29T16:24:57.545 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:24:40.678638967Z", 2020-11-29T16:24:57.545 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": { 2020-11-29T16:24:57.545 INFO:teuthology.orchestra.run.smithi168.stdout: "pool": 9, 2020-11-29T16:24:57.545 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 4 2020-11-29T16:24:57.546 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:57.546 INFO:teuthology.orchestra.run.smithi168.stdout: "bilog_flags": 0, 2020-11-29T16:24:57.546 INFO:teuthology.orchestra.run.smithi168.stdout: "versioned": false, 2020-11-29T16:24:57.546 INFO:teuthology.orchestra.run.smithi168.stdout: "owner": "", 2020-11-29T16:24:57.547 INFO:teuthology.orchestra.run.smithi168.stdout: "owner_display_name": "", 2020-11-29T16:24:57.547 INFO:teuthology.orchestra.run.smithi168.stdout: "zones_trace": [ 2020-11-29T16:24:57.547 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:57.547 INFO:teuthology.orchestra.run.smithi168.stdout: "entry": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-69:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.20" 2020-11-29T16:24:57.548 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:57.548 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:24:57.548 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:57.548 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:57.549 INFO:teuthology.orchestra.run.smithi168.stdout: "op_id": "5#00000000001.55.5", 2020-11-29T16:24:57.549 INFO:teuthology.orchestra.run.smithi168.stdout: "op_tag": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5482.5259", 2020-11-29T16:24:57.549 INFO:teuthology.orchestra.run.smithi168.stdout: "op": "write", 2020-11-29T16:24:57.550 INFO:teuthology.orchestra.run.smithi168.stdout: "object": "d", 2020-11-29T16:24:57.550 INFO:teuthology.orchestra.run.smithi168.stdout: "instance": "", 2020-11-29T16:24:57.550 INFO:teuthology.orchestra.run.smithi168.stdout: "state": "complete", 2020-11-29T16:24:57.550 INFO:teuthology.orchestra.run.smithi168.stdout: "index_ver": 1, 2020-11-29T16:24:57.551 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:24:40.827085832Z", 2020-11-29T16:24:57.551 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": { 2020-11-29T16:24:57.551 INFO:teuthology.orchestra.run.smithi168.stdout: "pool": 9, 2020-11-29T16:24:57.551 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 2 2020-11-29T16:24:57.552 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:57.552 INFO:teuthology.orchestra.run.smithi168.stdout: "bilog_flags": 0, 2020-11-29T16:24:57.552 INFO:teuthology.orchestra.run.smithi168.stdout: "versioned": false, 2020-11-29T16:24:57.552 INFO:teuthology.orchestra.run.smithi168.stdout: "owner": "", 2020-11-29T16:24:57.553 INFO:teuthology.orchestra.run.smithi168.stdout: "owner_display_name": "", 2020-11-29T16:24:57.553 INFO:teuthology.orchestra.run.smithi168.stdout: "zones_trace": [ 2020-11-29T16:24:57.553 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:57.553 INFO:teuthology.orchestra.run.smithi168.stdout: "entry": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-69:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.20" 2020-11-29T16:24:57.554 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:57.554 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:24:57.554 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:57.554 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:57.555 INFO:teuthology.orchestra.run.smithi168.stdout: "op_id": "8#00000000001.59.5", 2020-11-29T16:24:57.555 INFO:teuthology.orchestra.run.smithi168.stdout: "op_tag": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5482.5254", 2020-11-29T16:24:57.555 INFO:teuthology.orchestra.run.smithi168.stdout: "op": "write", 2020-11-29T16:24:57.555 INFO:teuthology.orchestra.run.smithi168.stdout: "object": "c", 2020-11-29T16:24:57.556 INFO:teuthology.orchestra.run.smithi168.stdout: "instance": "", 2020-11-29T16:24:57.556 INFO:teuthology.orchestra.run.smithi168.stdout: "state": "complete", 2020-11-29T16:24:57.556 INFO:teuthology.orchestra.run.smithi168.stdout: "index_ver": 1, 2020-11-29T16:24:57.556 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:24:40.778154815Z", 2020-11-29T16:24:57.557 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": { 2020-11-29T16:24:57.557 INFO:teuthology.orchestra.run.smithi168.stdout: "pool": 9, 2020-11-29T16:24:57.557 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 4 2020-11-29T16:24:57.557 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:57.558 INFO:teuthology.orchestra.run.smithi168.stdout: "bilog_flags": 0, 2020-11-29T16:24:57.558 INFO:teuthology.orchestra.run.smithi168.stdout: "versioned": false, 2020-11-29T16:24:57.558 INFO:teuthology.orchestra.run.smithi168.stdout: "owner": "", 2020-11-29T16:24:57.558 INFO:teuthology.orchestra.run.smithi168.stdout: "owner_display_name": "", 2020-11-29T16:24:57.559 INFO:teuthology.orchestra.run.smithi168.stdout: "zones_trace": [ 2020-11-29T16:24:57.559 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:57.559 INFO:teuthology.orchestra.run.smithi168.stdout: "entry": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-69:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.20" 2020-11-29T16:24:57.559 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:57.560 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:24:57.560 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:57.560 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:57.560 INFO:teuthology.orchestra.run.smithi168.stdout: "op_id": "10#00000000001.48.5", 2020-11-29T16:24:57.561 INFO:teuthology.orchestra.run.smithi168.stdout: "op_tag": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5482.5251", 2020-11-29T16:24:57.561 INFO:teuthology.orchestra.run.smithi168.stdout: "op": "write", 2020-11-29T16:24:57.561 INFO:teuthology.orchestra.run.smithi168.stdout: "object": "b", 2020-11-29T16:24:57.561 INFO:teuthology.orchestra.run.smithi168.stdout: "instance": "", 2020-11-29T16:24:57.562 INFO:teuthology.orchestra.run.smithi168.stdout: "state": "complete", 2020-11-29T16:24:57.562 INFO:teuthology.orchestra.run.smithi168.stdout: "index_ver": 1, 2020-11-29T16:24:57.562 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:24:40.725871065Z", 2020-11-29T16:24:57.563 INFO:teuthology.orchestra.run.smithi168.stdout: "ver": { 2020-11-29T16:24:57.563 INFO:teuthology.orchestra.run.smithi168.stdout: "pool": 9, 2020-11-29T16:24:57.563 INFO:teuthology.orchestra.run.smithi168.stdout: "epoch": 3 2020-11-29T16:24:57.563 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:57.564 INFO:teuthology.orchestra.run.smithi168.stdout: "bilog_flags": 0, 2020-11-29T16:24:57.564 INFO:teuthology.orchestra.run.smithi168.stdout: "versioned": false, 2020-11-29T16:24:57.564 INFO:teuthology.orchestra.run.smithi168.stdout: "owner": "", 2020-11-29T16:24:57.564 INFO:teuthology.orchestra.run.smithi168.stdout: "owner_display_name": "", 2020-11-29T16:24:57.565 INFO:teuthology.orchestra.run.smithi168.stdout: "zones_trace": [ 2020-11-29T16:24:57.565 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:57.565 INFO:teuthology.orchestra.run.smithi168.stdout: "entry": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76:kcmyep-69:a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.20" 2020-11-29T16:24:57.565 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:57.566 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:24:57.566 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:57.566 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:24:57.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:24:57.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:24:57.569 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:24:57.647 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:57.648 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:59.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'kcmyep-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:59.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'kcmyep-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:59.086 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket kcmyep-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:59.167 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:59.168 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:59.438 INFO:teuthology.orchestra.run.smithi168.stdout:[] 2020-11-29T16:24:59.441 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2020-11-29T16:24:59.481 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-11-29T16:24:59.482 INFO:tasks.rgw_multisite_tests: 2020-11-29T16:24:59.483 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-11-29T16:24:59.484 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1513.111s 2020-11-29T16:24:59.484 INFO:tasks.rgw_multisite_tests: 2020-11-29T16:24:59.485 INFO:tasks.rgw_multisite_tests:OK 2020-11-29T16:24:59.489 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-11-29T16:24:59.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:59.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:59.492 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:59.532 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:59.532 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:24:59.768 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:24:59.809 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:24:59.809 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:59.809 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:59.810 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:59.810 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:59.810 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:59.811 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:24:59.811 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:24:48.760284Z" 2020-11-29T16:24:59.811 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:59.811 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:59.812 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:59.812 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:59.812 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:24:59.812 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:24:59.812 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:24:59.813 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:24:59.813 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:24:59.813 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:24:59.814 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:24:59.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:59.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:24:59.815 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:24:59.865 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:24:59.865 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:00.099 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:00.099 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:00.100 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:00.101 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:00.102 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:00.103 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:00.105 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:00.106 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:25:00.106 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:00.106 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:00.106 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:00.107 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:48.760284Z", 2020-11-29T16:25:00.107 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:00.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.108 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:00.108 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:00.108 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:00.109 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:00.109 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:00.109 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:00.109 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:00.110 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:00.110 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:00.110 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:00.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.111 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:00.111 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:00.112 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:00.112 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:00.112 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:00.112 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:00.113 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:00.113 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:00.113 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:00.113 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:00.114 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.114 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.114 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:00.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.115 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:00.115 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:00.115 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:00.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.116 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:00.118 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:00.118 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:00.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:00.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:00.120 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:00.162 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:00.162 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:00.428 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:00.428 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:00.429 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:00.429 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:00.429 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:00.429 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:00.430 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:00.430 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:00.430 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:00.430 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:00.431 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:00.431 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:00.431 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:00.431 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:00.432 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:00.432 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:00.432 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:00.433 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:00.433 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:00.433 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:00.434 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:00.434 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:00.434 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:00.434 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:00.435 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:00.435 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:25:00.435 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:00.435 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:00.436 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:00.436 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:24:48.760284Z", 2020-11-29T16:25:00.436 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:00.437 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:00.437 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:00.438 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:00.438 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:00.438 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:00.438 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:00.439 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:00.439 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:00.439 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:00.439 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:00.440 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:00.440 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:00.440 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:00.440 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:00.441 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:00.441 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:00.441 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:00.441 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:00.442 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:00.442 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:00.442 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:00.442 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:00.443 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:00.443 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:00.443 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:00.444 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:00.444 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:00.445 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:00.445 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:00.445 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:00.445 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:00.446 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:00.446 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:00.447 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:00.447 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:00.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:00.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:00.449 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:00.489 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:00.489 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:00.867 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:00.868 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:00.869 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:00.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:25:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:00.873 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:48.760284Z", 2020-11-29T16:25:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:00.875 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:00.876 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:00.878 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:00.879 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:00.880 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:00.881 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:00.882 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:00.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:00.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:00.884 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:00.926 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:00.927 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:01.167 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:01.202 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:01.203 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:01.203 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:01.203 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:01.203 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:01.204 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:01.204 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:25:01.204 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:24:48.760284Z" 2020-11-29T16:25:01.204 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:01.204 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:01.205 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:01.205 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:01.205 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:01.205 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:01.205 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:01.206 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:01.206 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:01.206 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:01.207 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:01.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:01.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:01.209 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:01.250 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:01.250 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:01.521 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:01.522 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:01.522 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:01.522 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:01.523 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:01.523 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:01.523 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:01.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:01.523 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:01.524 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:01.524 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:01.524 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:01.524 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:01.525 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:01.525 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:01.525 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:01.525 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:01.526 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:01.526 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:01.526 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:01.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:01.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:01.527 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:01.527 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:01.527 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:01.527 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667088.760284_486.1", 2020-11-29T16:25:01.528 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:01.528 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:01.528 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:01.528 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:24:48.760284Z", 2020-11-29T16:25:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:01.529 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:01.530 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:01.532 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:01.532 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:01.532 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:01.532 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:01.533 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:01.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:01.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:01.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:01.535 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:01.535 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:01.536 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:01.536 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:01.536 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:01.536 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:01.537 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:01.537 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:01.537 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:01.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:01.539 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:01.540 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:01.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:01.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:01.593 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:01.630 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:01.630 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:01.874 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:01.908 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:01.909 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:01.909 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:01.909 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:01.909 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:01.909 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:01.910 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667101.583397_490.1", 2020-11-29T16:25:01.910 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:01.583397Z" 2020-11-29T16:25:01.910 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:01.910 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:01.911 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:01.911 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:01.911 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:01.911 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:01.912 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:01.912 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:01.912 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:01.912 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:01.913 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:01.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:01.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:01.915 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:01.952 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:01.953 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:02.325 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:02.326 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:02.327 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:02.327 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:02.327 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:02.328 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:02.328 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:02.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:02.329 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:02.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:02.331 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:02.331 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:02.332 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:02.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:02.333 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:02.334 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:02.334 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:02.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:02.335 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:02.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:02.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:02.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:02.337 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:02.337 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:02.338 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:02.338 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667101.583397_490.1", 2020-11-29T16:25:02.338 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:02.339 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:02.340 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:01.583397Z", 2020-11-29T16:25:02.340 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:02.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:02.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:02.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:02.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:02.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:02.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:02.343 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:02.343 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:02.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:02.345 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:02.345 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:02.345 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:02.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:02.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:02.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:02.346 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:02.346 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:02.347 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:02.347 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:02.347 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:02.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:02.348 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:02.348 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:02.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:02.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:02.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:02.349 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:02.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:02.350 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:02.350 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:02.350 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:02.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:02.351 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:02.352 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:02.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:02.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:02.598 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:02.638 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:02.638 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:02.911 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:02.953 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:02.955 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:02.956 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:02.956 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:02.956 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:02.957 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:02.958 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667101.583397_490.1", 2020-11-29T16:25:02.962 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:01.583397Z" 2020-11-29T16:25:02.962 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:02.962 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:02.962 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:02.963 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:02.963 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:02.963 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:02.963 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:02.964 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:02.964 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:02.964 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:02.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:02.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:02.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:02.967 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:03.017 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:03.019 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:03.275 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:03.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:03.277 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:03.278 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:03.278 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:03.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:03.279 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:03.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:03.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:03.279 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:03.280 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:03.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:03.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:03.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:03.280 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:03.281 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:03.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:03.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:03.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:03.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:03.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:03.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:03.282 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:03.283 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667101.583397_490.1", 2020-11-29T16:25:03.283 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:03.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:03.283 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:03.283 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:01.583397Z", 2020-11-29T16:25:03.283 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:03.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:03.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:03.284 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:03.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:03.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:03.285 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:03.285 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:03.285 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:03.285 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:03.285 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:03.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:03.286 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:03.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:03.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:03.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:03.287 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:03.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:03.287 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:03.287 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:03.288 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:03.288 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:03.288 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:03.288 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:03.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:03.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:03.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:03.289 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:03.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:03.290 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:03.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:03.290 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:03.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:03.290 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:03.291 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:03.441 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2020-11-29T16:25:03.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:03.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:03.443 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:03.481 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:03.481 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:03.728 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:03.755 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:03.755 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:03.755 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:03.756 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:03.756 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:03.756 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:03.756 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667103.433406_492.1", 2020-11-29T16:25:03.757 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:03.433406Z" 2020-11-29T16:25:03.757 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:03.757 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:03.757 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:03.758 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:03.758 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:03.758 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:03.759 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:03.759 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:03.759 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:03.759 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:03.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:03.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:03.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:03.762 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:03.804 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:03.805 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:04.063 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:04.064 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:04.064 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:04.065 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:04.065 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:04.065 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:04.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:04.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.066 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:04.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:04.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:04.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:04.067 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:04.067 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:04.067 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:04.067 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:04.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:04.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:04.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:04.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:04.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:04.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:04.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:04.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667103.433406_492.1", 2020-11-29T16:25:04.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:04.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:04.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:04.070 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:03.433406Z", 2020-11-29T16:25:04.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:04.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:04.071 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:04.072 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:04.072 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:04.072 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:04.072 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:04.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:04.073 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:04.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:04.074 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:04.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:04.075 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:04.075 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:04.076 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:04.076 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:04.076 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:04.076 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:04.077 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:04.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:04.077 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:04.077 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.078 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:04.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.078 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:04.079 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:04.079 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:04.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.079 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:04.081 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:04.081 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:04.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:04.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:04.083 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:04.124 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:04.125 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:04.398 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:04.399 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:04.399 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:04.399 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:04.399 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:04.400 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:04.400 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:04.400 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:04.401 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:04.401 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:04.401 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:04.401 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:04.402 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:04.402 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:04.403 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:04.404 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:04.406 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:04.406 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:04.407 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:04.407 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:04.407 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:04.407 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:04.408 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:04.408 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:04.408 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:04.408 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667103.433406_492.1", 2020-11-29T16:25:04.409 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:04.409 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:04.410 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:04.410 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:03.433406Z", 2020-11-29T16:25:04.410 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:04.410 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:04.411 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:04.411 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:04.411 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:04.412 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:04.412 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:04.412 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:04.412 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:04.412 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:04.413 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:04.413 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:04.413 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:04.413 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:04.413 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:04.414 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:04.414 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:04.414 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:04.414 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:04.415 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:04.415 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:04.415 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:04.415 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:04.415 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:04.416 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:04.416 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:04.416 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:04.416 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:04.416 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:04.417 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:04.417 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:04.417 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:04.417 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:04.417 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:04.418 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:04.419 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:04.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:04.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:04.420 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:04.462 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:04.463 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:04.733 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:04.733 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:04.733 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:04.734 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:04.734 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:04.734 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:04.734 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:04.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.735 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:04.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:04.735 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:04.735 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:04.736 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:04.736 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:04.736 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:04.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:04.736 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:04.737 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:04.737 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:04.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:04.738 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:04.738 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:04.738 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:04.738 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667103.433406_492.1", 2020-11-29T16:25:04.738 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:04.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:04.739 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:04.739 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:03.433406Z", 2020-11-29T16:25:04.739 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:04.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:04.740 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:04.740 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:04.740 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:04.741 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:04.741 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:04.741 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:04.741 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:04.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:04.742 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:04.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:04.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:04.743 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:04.743 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:04.743 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:04.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:04.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:04.744 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:04.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:04.744 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:04.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:04.745 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:04.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:04.746 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:04.747 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:04.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:04.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:04.748 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:04.789 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:04.790 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:05.041 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:05.085 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:05.085 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:05.085 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:05.085 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:05.086 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:05.086 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:05.086 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667103.433406_492.1", 2020-11-29T16:25:05.086 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:03.433406Z" 2020-11-29T16:25:05.087 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:05.087 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:05.087 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:05.087 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:05.087 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:05.088 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:05.088 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:05.088 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:05.088 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:05.088 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:05.089 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:05.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:05.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:05.091 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:05.133 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:05.134 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:05.508 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:05.508 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:05.508 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:05.509 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:05.509 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:05.509 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:05.509 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:05.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:05.510 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:05.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:05.510 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:05.510 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:05.511 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:05.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:05.511 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:05.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:05.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:05.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:05.512 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:05.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:05.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:05.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:05.513 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:05.513 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:05.513 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:05.514 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667103.433406_492.1", 2020-11-29T16:25:05.514 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:05.514 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:05.514 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:05.515 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:03.433406Z", 2020-11-29T16:25:05.515 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:05.515 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:05.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:05.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:05.516 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:05.516 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:05.516 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:05.516 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:05.517 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:05.517 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:05.517 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:05.517 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:05.517 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:05.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:05.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:05.518 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:05.518 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:05.518 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:05.519 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:05.519 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:05.519 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:05.519 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:05.519 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:05.520 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:05.520 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:05.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:05.520 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:05.520 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:05.521 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:05.521 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:05.521 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:05.521 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:05.521 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:05.522 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:05.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:05.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:05.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:05.576 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:05.613 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:05.613 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:05.845 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:05.881 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:05.884 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:05.886 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:05.888 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:05.890 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:05.892 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:05.894 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667105.566129_496.1", 2020-11-29T16:25:05.896 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:05.566129Z" 2020-11-29T16:25:05.896 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:05.897 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:05.897 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:05.898 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:05.898 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:05.900 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:05.901 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:05.902 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:05.902 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:05.902 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:05.903 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:05.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:05.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:05.905 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:05.947 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:05.947 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:06.220 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:06.220 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:06.221 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:06.221 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:06.221 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:06.221 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:06.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:06.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.222 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:06.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:06.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:06.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:06.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:06.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:06.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:06.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:06.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:06.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:06.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:06.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:06.226 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:06.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:06.227 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:06.227 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667105.566129_496.1", 2020-11-29T16:25:06.227 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:06.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:06.228 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:06.228 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:05.566129Z", 2020-11-29T16:25:06.229 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:06.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:06.230 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:06.230 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.234 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:06.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.239 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:06.239 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:06.239 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:06.240 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.240 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:06.241 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:06.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:06.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:06.290 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:06.331 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:06.331 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:06.579 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:06.616 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:06.616 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:06.616 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:06.617 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:06.617 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:06.617 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:06.617 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667105.566129_496.1", 2020-11-29T16:25:06.618 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:05.566129Z" 2020-11-29T16:25:06.618 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:06.618 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:06.618 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:06.619 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:06.619 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:06.619 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:06.619 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:06.620 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:06.620 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:06.620 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:06.621 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:06.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:06.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:06.623 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:06.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:06.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:06.934 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:06.935 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:06.935 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:06.935 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:06.936 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:06.936 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:06.937 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:06.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.938 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:06.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:06.939 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:06.939 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:06.939 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:06.940 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:06.940 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:06.941 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:06.941 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:06.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:06.942 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:06.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.944 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:06.944 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:06.945 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:06.945 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:06.946 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667105.566129_496.1", 2020-11-29T16:25:06.946 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:06.947 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:06.947 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:06.947 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:05.566129Z", 2020-11-29T16:25:06.948 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:06.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:06.950 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:06.950 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:06.950 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:06.951 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:06.951 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:06.952 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:06.952 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:06.952 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:06.953 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:06.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:06.954 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:06.955 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:06.955 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:06.955 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:06.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:06.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:06.957 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:06.957 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:06.958 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:06.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.959 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:06.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:06.959 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:06.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:06.960 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:06.960 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:06.961 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:06.962 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:07.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:25:07.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-72', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:25:07.412 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-72 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:25:07.470 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:07.471 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:07.692 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:07.683+0000 7f323ed51080 1 bucket sync caught up with source: 2020-11-29T16:25:07.694 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.91.5, , 00000000001.92.5, 00000000001.80.5, 00000000001.76.5, , 00000000001.62.5, 00000000001.121.5, 00000000001.73.5, 00000000002.77.5] 2020-11-29T16:25:07.694 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.91.5, , 00000000001.92.5, 00000000001.80.5, 00000000001.76.5, , 00000000001.62.5, 00000000001.121.5, 00000000001.73.5, 00000000002.77.5] 2020-11-29T16:25:07.694 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:07.683+0000 7f323ed51080 0 bucket checkpoint complete 2020-11-29T16:25:07.737 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:07.031084Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '2', 'size': 0, 'etag': '586e3bab', 'versionId': '', 'sequencer': '63CBC35F9035F40B', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.200553.586e3bab', 'opaqueData': ''} 2020-11-29T16:25:07.738 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:07.078690Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '3', 'size': 0, 'etag': 'e65279ff', 'versionId': '', 'sequencer': '63CBC35F0A3EFB0B', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.201014.e65279ff', 'opaqueData': ''} 2020-11-29T16:25:07.738 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:06.987407Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '1', 'size': 0, 'etag': '2d1ca5c4', 'versionId': '', 'sequencer': '63CBC35FBD6D6C0E', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.241987.2d1ca5c4', 'opaqueData': ''} 2020-11-29T16:25:07.739 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:06.967976Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '0', 'size': 0, 'etag': '3739fae7', 'versionId': '', 'sequencer': '63CBC35FF33A0311', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.285424.3739fae7', 'opaqueData': ''} 2020-11-29T16:25:07.739 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:07.174592Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '5', 'size': 0, 'etag': 'db27cb69', 'versionId': '', 'sequencer': '63CBC35F5F861F17', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.387941.db27cb69', 'opaqueData': ''} 2020-11-29T16:25:07.739 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:07.270457Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '7', 'size': 0, 'etag': 'd84dacf3', 'versionId': '', 'sequencer': '63CBC35F1817D719', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.433526.d84dacf3', 'opaqueData': ''} 2020-11-29T16:25:07.740 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:07.222807Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '6', 'size': 0, 'etag': '1ae7712b', 'versionId': '', 'sequencer': '63CBC35FA831E019', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.434123.1ae7712b', 'opaqueData': ''} 2020-11-29T16:25:07.740 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:07.126884Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '4', 'size': 0, 'etag': 'fa89eb11', 'versionId': '', 'sequencer': '63CBC35F812FAF1C', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.481243.fa89eb11', 'opaqueData': ''} 2020-11-29T16:25:07.741 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:07.366322Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '9', 'size': 0, 'etag': '34eaf78f', 'versionId': '', 'sequencer': '63CBC35F36C2B31C', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.481542.34eaf78f', 'opaqueData': ''} 2020-11-29T16:25:07.741 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-11-29T16:25:07.318264Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'kcmyep-72_notif', 'bucket': {'name': 'kcmyep-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::kcmyep-72', 'id': 'a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.23'}, 'object': {'key': '8', 'size': 0, 'etag': 'a28c1d81', 'versionId': '', 'sequencer': '63CBC35FD20ABA1C', 'metadata': [], 'tags': []}}, 'eventId': '1606667107.481954.a28c1d81', 'opaqueData': ''} 2020-11-29T16:25:07.751 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2020-11-29T16:25:07.947 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2020-11-29T16:25:07.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:07.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:07.949 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:07.998 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:07.999 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:08.250 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:08.288 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:08.288 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:08.289 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:08.289 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:08.289 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:08.289 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:08.289 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667107.938515_501.1", 2020-11-29T16:25:08.290 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:07.938515Z" 2020-11-29T16:25:08.290 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:08.290 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:08.290 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:08.291 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:08.291 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:08.291 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:08.291 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:08.291 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:08.292 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:08.292 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:08.293 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:08.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:08.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:08.294 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:08.367 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:08.367 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:08.639 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:08.640 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:08.640 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:08.641 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:08.642 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:08.642 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:08.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:08.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:08.644 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:08.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:08.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:08.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:08.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:08.646 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:08.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:08.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:08.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:08.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:08.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:08.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:08.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:08.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:08.650 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:08.650 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:08.651 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:08.651 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667107.938515_501.1", 2020-11-29T16:25:08.652 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:08.652 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:08.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:08.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:07.938515Z", 2020-11-29T16:25:08.653 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:08.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:08.655 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:08.656 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:08.657 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:08.657 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:08.658 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:08.658 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:08.658 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:08.659 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:08.659 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:08.660 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:08.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:08.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:08.661 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:08.661 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:08.662 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:08.662 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:08.662 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:08.663 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:08.663 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:08.663 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:08.663 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:08.664 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:08.664 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:08.665 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:08.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:08.665 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:08.665 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:08.666 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:08.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:08.666 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:08.667 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:08.668 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:08.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:08.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:08.669 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:08.732 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:08.732 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:09.012 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:09.013 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:09.014 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:09.015 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:09.016 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:09.016 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:09.017 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:09.017 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:09.018 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:09.019 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:09.019 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:09.019 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:09.020 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:09.020 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:09.020 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:09.021 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:09.021 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:09.021 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:09.022 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:09.022 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:09.022 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:09.023 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:09.023 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:09.023 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:09.024 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:09.024 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667107.938515_501.1", 2020-11-29T16:25:09.024 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:09.025 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:09.025 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:09.025 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:07.938515Z", 2020-11-29T16:25:09.026 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:09.030 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:09.031 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:09.031 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:09.031 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:09.031 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:09.032 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:09.032 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:09.032 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:09.032 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:09.032 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:09.033 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:09.033 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:09.033 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:09.033 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:09.034 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:09.034 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:09.034 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:09.034 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:09.034 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:09.035 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:09.035 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:09.035 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:09.035 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:09.036 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:09.036 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:09.037 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:09.037 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:09.037 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:09.038 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:09.038 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:09.038 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:09.039 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:09.039 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:09.040 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:09.041 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:09.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:09.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:09.043 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:09.091 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:09.460 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:09.461 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:09.461 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:09.461 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:09.461 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:09.462 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:09.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:09.463 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:09.463 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:09.463 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:09.463 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:09.464 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:09.464 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:09.464 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:09.464 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:09.464 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:09.465 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:09.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:09.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:09.465 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:09.466 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:09.466 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:09.466 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667107.938515_501.1", 2020-11-29T16:25:09.466 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:09.466 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:09.467 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:09.467 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:07.938515Z", 2020-11-29T16:25:09.467 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:09.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:09.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:09.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:09.468 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:09.468 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:09.468 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:09.469 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:09.469 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:09.469 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:09.469 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:09.469 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:09.470 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:09.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:09.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:09.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:09.471 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:09.471 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:09.471 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:09.471 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:09.471 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:09.472 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:09.472 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:09.472 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:09.472 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:09.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:09.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:09.473 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:09.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:09.473 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:09.473 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:09.474 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:09.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:09.474 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:09.475 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:09.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:09.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:09.477 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:09.520 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:09.520 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:09.762 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:09.796 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:09.796 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:09.797 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:09.797 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:09.797 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:09.797 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:09.798 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667107.938515_501.1", 2020-11-29T16:25:09.798 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:07.938515Z" 2020-11-29T16:25:09.798 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:09.800 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:09.800 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:09.800 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:09.800 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:09.800 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:09.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:09.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:09.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:09.803 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:09.844 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:09.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:10.214 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:10.215 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:10.215 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:10.215 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:10.215 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667107.938515_501.1", 2020-11-29T16:25:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:07.938515Z", 2020-11-29T16:25:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:10.225 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:10.225 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:10.225 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:10.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:10.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:10.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:10.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.228 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:10.229 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:10.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:10.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:10.275 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:10.315 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:10.315 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:10.550 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:10.586 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:10.587 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:10.588 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:10.588 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667110.267515_505.1", 2020-11-29T16:25:10.588 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:10.267515Z" 2020-11-29T16:25:10.588 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:10.589 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:10.590 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:10.590 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:10.590 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:10.590 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:10.591 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:10.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:10.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:10.593 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:10.633 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:10.634 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:10.907 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:10.909 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:10.910 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:10.911 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:10.912 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:10.914 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:10.914 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:10.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.917 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:10.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:10.919 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:10.919 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:10.919 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:10.920 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:10.920 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:10.921 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:10.921 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:10.922 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:10.922 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:10.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:10.923 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:10.923 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:10.923 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:10.924 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667110.267515_505.1", 2020-11-29T16:25:10.924 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:10.924 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:10.925 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:10.925 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:10.267515Z", 2020-11-29T16:25:10.925 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:10.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:10.927 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:10.927 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:10.928 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:10.928 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:10.928 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:10.928 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:10.929 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:10.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:10.929 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:10.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:10.930 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:10.930 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:10.930 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:10.930 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:10.931 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:10.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:10.931 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:10.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:10.931 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:10.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.932 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:10.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:10.933 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:10.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:10.933 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:10.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:10.933 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:10.935 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:11.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:11.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:11.013 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:11.053 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:11.053 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:11.297 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:11.336 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:11.336 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:11.337 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:11.337 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:11.337 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:11.337 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:11.338 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667110.267515_505.1", 2020-11-29T16:25:11.338 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:10.267515Z" 2020-11-29T16:25:11.338 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:11.338 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:11.339 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:11.339 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:11.339 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:11.339 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:11.339 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:11.340 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:11.340 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:11.340 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:11.341 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:11.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:11.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:11.343 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:11.384 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:11.385 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:11.647 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:11.647 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:11.648 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:11.649 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:11.650 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:11.652 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:11.652 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:11.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:11.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667110.267515_505.1", 2020-11-29T16:25:11.656 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:11.656 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:11.656 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:11.657 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:10.267515Z", 2020-11-29T16:25:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:11.658 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:11.659 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:11.659 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:11.659 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:11.659 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:11.660 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:11.661 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:11.662 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:11.663 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:11.664 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:11.665 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:11.666 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:11.821 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2020-11-29T16:25:11.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:11.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:11.823 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:11.860 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:11.861 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:12.099 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:12.140 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:12.140 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:12.141 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:12.141 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:12.141 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:12.141 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:12.142 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:12.142 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:11.812389Z" 2020-11-29T16:25:12.142 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:12.142 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:12.143 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:12.143 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:12.143 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:12.143 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:12.144 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:12.144 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:12.144 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:12.144 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:12.145 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:12.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:12.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:12.147 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:12.188 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:12.189 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:12.471 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:12.471 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:12.471 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:12.471 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:12.472 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:12.472 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:12.472 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:12.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:12.473 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:12.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:12.473 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:12.473 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:12.474 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:12.474 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:12.474 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:12.474 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:12.474 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:12.475 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:12.475 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:12.475 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:12.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:12.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:12.476 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:12.476 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:12.476 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:12.476 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:12.477 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:12.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:12.477 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:12.477 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:12.478 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:12.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:12.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:12.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:12.479 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:12.479 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:12.479 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:12.479 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:12.480 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:12.480 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:12.480 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:12.480 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:12.481 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:12.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:12.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:12.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:12.482 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:12.482 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:12.482 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:12.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:12.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:12.483 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:12.483 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:12.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:12.483 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:12.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:12.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:12.484 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:12.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:12.485 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:12.485 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:12.485 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:12.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:12.486 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:12.487 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:12.487 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:12.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:12.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:12.489 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:12.529 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:12.530 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:12.811 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:12.811 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:12.812 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:12.813 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:12.814 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:12.814 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:12.814 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:12.814 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:12.815 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:12.816 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:12.817 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:12.818 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:12.819 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:12.820 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:12.821 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:12.822 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:12.823 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:12.824 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:12.825 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:12.825 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:12.826 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:12.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:12.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:12.827 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:12.869 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:12.870 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:13.235 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:13.235 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:13.235 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:13.236 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:13.236 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:13.236 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:13.236 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:13.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.237 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:13.237 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:13.237 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:13.237 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:13.238 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:13.238 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:13.238 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:13.238 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:13.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:13.240 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:13.241 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:13.241 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:13.241 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:13.241 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:13.242 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:13.242 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:13.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:13.243 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:13.243 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:13.243 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:13.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:13.243 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:13.244 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:13.244 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:13.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:13.244 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:13.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.245 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:13.245 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:13.245 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:13.245 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:13.246 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:13.246 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:13.246 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:13.246 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:13.246 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:13.247 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:13.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.247 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:13.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.248 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:13.248 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:13.248 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:13.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.248 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:13.249 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:13.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:13.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:13.251 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:13.293 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:13.293 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:13.552 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:13.592 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:13.592 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:13.592 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:13.592 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:13.593 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:13.593 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:13.593 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:13.593 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:11.812389Z" 2020-11-29T16:25:13.593 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:13.594 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:13.594 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:13.594 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:13.595 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:13.595 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:13.595 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:13.595 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:13.595 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:13.596 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:13.597 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:13.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:13.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:13.599 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:13.639 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:13.639 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:13.902 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:13.902 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:13.903 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:13.904 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:13.905 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:13.906 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:13.907 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:13.908 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:13.910 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:13.911 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:13.912 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:13.913 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:13.914 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:13.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:13.916 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:13.917 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:13.918 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:14.055 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2020-11-29T16:25:14.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:14.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:14.057 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:14.098 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:14.099 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:14.339 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:14.367 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:14.368 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:14.368 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:14.368 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:14.369 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:14.369 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:14.369 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:14.369 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:11.812389Z" 2020-11-29T16:25:14.370 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:14.370 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:14.370 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:14.371 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:14.371 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:14.371 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:14.371 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:14.372 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:14.372 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:14.372 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:14.374 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:14.375 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:14.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:14.376 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:14.417 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:14.418 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:14.696 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:14.697 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:14.697 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:14.697 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:14.697 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:14.698 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:14.698 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:14.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:14.698 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:14.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:14.699 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:14.699 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:14.699 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:14.699 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:14.700 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:14.700 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:14.700 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:14.700 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:14.701 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:14.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:14.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:14.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:14.701 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:14.702 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:14.702 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:14.702 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:14.702 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:14.702 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:14.703 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:14.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:14.703 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:14.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:14.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:14.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:14.704 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:14.705 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:14.705 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:14.705 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:14.705 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:14.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:14.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:14.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:14.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:14.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:14.710 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:14.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:14.711 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:14.712 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:14.712 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:14.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:14.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:14.714 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:14.755 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:14.756 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:15.051 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:15.052 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:15.052 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:15.053 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:15.053 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:15.053 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:15.054 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:15.054 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:15.055 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:15.055 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:15.056 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:15.056 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:15.057 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:15.057 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:15.058 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:15.058 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:15.058 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:15.058 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:15.059 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:15.059 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:15.059 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:15.059 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:15.059 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:15.060 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:15.060 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:15.060 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:15.060 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:15.061 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:15.061 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:15.061 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:15.061 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:15.061 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:15.062 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:15.062 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:15.062 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:15.062 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:15.062 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:15.063 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:15.063 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:15.063 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:15.063 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:15.064 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:15.064 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:15.064 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:15.064 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:15.064 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:15.065 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:15.065 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:15.065 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:15.065 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:15.065 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:15.066 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:15.066 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:15.066 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:15.066 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:15.067 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:15.067 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:15.067 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:15.068 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:15.068 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:15.068 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:15.068 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:15.069 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:15.069 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:15.070 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:15.074 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:15.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:15.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:15.076 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:15.118 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:15.119 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:15.388 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:15.388 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:15.389 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:15.390 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:15.391 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:15.392 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:15.393 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:15.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:15.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:15.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:15.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:15.400 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:15.401 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:15.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:15.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:15.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:15.401 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:15.402 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:15.402 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:15.402 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:15.402 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:15.402 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:15.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:15.404 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:15.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:15.404 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:15.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:15.404 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:15.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:15.405 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:15.406 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:15.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:15.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:15.407 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:15.447 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:15.447 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:15.696 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:15.733 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:15.734 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:15.734 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:15.734 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:15.734 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:15.735 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:15.735 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:15.735 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:11.812389Z" 2020-11-29T16:25:15.735 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:15.736 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:15.736 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:15.736 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:15.736 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:15.736 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:15.737 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:15.737 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:15.737 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:15.737 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:15.738 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:15.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:15.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:15.740 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:15.783 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:15.783 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:16.155 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:16.155 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:16.156 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:16.157 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:16.157 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:16.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.157 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:16.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:16.158 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:16.158 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:16.158 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:16.159 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:16.159 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:16.159 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:16.160 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:16.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:16.160 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:16.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:16.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:16.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:16.161 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:16.162 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:16.162 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:16.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:16.162 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:16.162 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:16.163 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:16.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:16.164 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:16.164 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:16.164 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:16.164 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:16.164 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:16.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:16.166 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:16.166 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:16.167 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:16.167 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:16.167 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:16.167 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:16.167 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:16.168 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:16.168 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:16.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.168 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:16.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.169 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:16.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:16.169 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:16.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.170 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:16.171 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:16.171 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2020-11-29T16:25:16.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:16.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:16.173 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:16.213 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:16.214 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:16.446 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:16.481 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:16.481 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:16.482 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:16.482 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:16.482 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:16.483 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:16.483 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:16.483 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:11.812389Z" 2020-11-29T16:25:16.483 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:16.483 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:16.484 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:16.484 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:16.484 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:16.485 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:16.485 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:16.485 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:16.485 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:16.486 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:16.487 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:16.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:16.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:16.489 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:16.532 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:16.532 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:16.830 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:16.831 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:16.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:16.833 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:16.834 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:16.835 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:16.835 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:16.835 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:16.835 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:16.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:16.837 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:16.837 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:16.838 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:16.838 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:16.838 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:16.839 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:16.839 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:16.839 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:16.840 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:16.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:16.841 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:16.841 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:16.842 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:16.842 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:16.842 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:16.843 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:16.843 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:16.843 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:16.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:16.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:16.846 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:16.846 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:16.846 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:16.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:16.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:16.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:16.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:16.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:16.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:16.851 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:16.852 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:16.852 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:16.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:16.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:16.854 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:16.895 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:16.895 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:17.177 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:17.177 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:17.178 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:17.178 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:17.178 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:17.179 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:17.179 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:17.180 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:17.180 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:17.180 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:17.181 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:17.181 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:17.181 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:17.181 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:17.182 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:17.182 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:17.182 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:17.182 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:17.183 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:17.183 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:17.183 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:17.183 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:17.184 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:17.184 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:17.184 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:17.184 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:17.184 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:17.185 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:17.185 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:17.185 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:17.185 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:17.186 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:17.186 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:17.186 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:17.186 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:17.187 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:17.187 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:17.187 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:17.187 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:17.187 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:17.188 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:17.188 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:17.188 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:17.188 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:17.189 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:17.189 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:17.189 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:17.189 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:17.189 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:17.190 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:17.190 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:17.190 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:17.190 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:17.191 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:17.191 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:17.191 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:17.191 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:17.191 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:17.192 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:17.192 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:17.192 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:17.192 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:17.193 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:17.193 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:17.194 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:17.195 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:17.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:17.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:17.196 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:17.240 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:17.240 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:17.602 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:17.602 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:17.603 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:17.603 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:17.603 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:17.603 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:17.604 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:17.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:17.604 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:17.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:17.604 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:17.605 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:17.605 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:17.605 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:17.605 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:17.606 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:17.607 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:17.607 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:17.607 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:17.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:17.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:17.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:17.608 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:17.608 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:17.608 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:17.609 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:17.609 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:17.609 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:17.609 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:17.610 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:17.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:17.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:17.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:17.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:17.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:17.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:17.611 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:17.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:17.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:17.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:17.612 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:17.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:17.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:17.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:17.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:17.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:17.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:17.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:17.614 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:17.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:17.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:17.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:17.616 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:17.616 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:17.616 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:17.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:17.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:17.618 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:17.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:17.619 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:17.619 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:17.619 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:17.619 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:17.620 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:17.621 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:17.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:17.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:17.622 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:17.661 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:17.662 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:17.905 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:17.942 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:17.943 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:17.944 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:17.944 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:17.944 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:17.945 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:17.945 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:17.946 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:11.812389Z" 2020-11-29T16:25:17.946 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:17.947 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:17.950 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:17.950 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:17.951 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:17.951 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:17.951 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:17.951 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:17.952 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:17.952 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:17.953 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:17.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:17.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:17.955 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:17.995 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:17.995 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:18.360 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:18.361 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:18.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:18.363 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:18.363 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:18.363 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:18.363 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:18.363 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:18.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:18.364 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:18.364 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:18.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:18.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:18.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:18.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:18.365 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:18.365 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:18.365 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:18.366 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667111.812389_507.1", 2020-11-29T16:25:18.366 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:18.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:18.366 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:18.367 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:11.812389Z", 2020-11-29T16:25:18.367 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:18.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:18.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:18.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:18.368 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:18.368 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:18.368 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:18.368 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:18.368 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:18.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:18.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:18.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:18.370 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:18.370 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:18.371 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:18.371 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:18.371 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:18.371 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:18.371 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:18.372 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:18.372 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:18.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:18.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:18.372 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:18.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:18.373 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:18.373 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:18.373 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:18.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:18.374 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:18.374 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:18.727 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2020-11-29T16:25:18.728 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2020-11-29T16:25:18.728 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2020-11-29T16:25:18.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:18.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:18.730 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:18.766 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:18.766 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:19.003 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:19.033 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:19.034 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:19.034 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:19.034 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:19.035 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:19.035 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:19.035 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667118.715630_513.1", 2020-11-29T16:25:19.035 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:18.715630Z" 2020-11-29T16:25:19.035 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:19.036 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:19.036 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:19.036 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:19.036 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:19.037 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:19.037 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:19.037 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:19.037 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:19.037 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:19.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:19.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:19.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:19.042 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:19.085 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:19.085 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:19.387 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:19.387 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:19.388 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:19.388 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:19.388 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:19.388 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:19.388 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:19.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:19.389 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:19.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:19.389 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:19.389 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:19.390 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:19.390 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:19.390 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:19.390 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:19.391 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:19.391 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:19.391 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:19.391 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:19.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:19.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:19.392 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:19.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:19.392 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:19.392 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667118.715630_513.1", 2020-11-29T16:25:19.393 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:19.393 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:19.393 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:19.393 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:18.715630Z", 2020-11-29T16:25:19.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:19.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:19.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:19.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:19.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:19.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:19.395 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:19.395 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:19.395 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:19.395 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:19.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:19.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:19.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:19.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:19.397 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:19.397 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:19.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:19.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:19.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:19.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:19.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:19.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:19.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:19.399 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:19.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:19.400 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:19.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:19.400 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:19.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:19.400 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:19.402 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:19.402 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:19.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:19.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:19.404 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:19.438 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:19.438 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:19.700 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:19.701 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:19.702 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:19.703 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:19.704 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:19.705 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667118.715630_513.1", 2020-11-29T16:25:19.706 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:18.715630Z", 2020-11-29T16:25:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:19.707 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:19.708 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:19.709 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:19.710 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:19.711 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:19.712 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:19.713 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:19.714 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:19.714 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:19.715 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:19.716 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:19.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:19.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:19.718 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:19.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:19.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:20.035 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:20.035 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:20.035 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:20.035 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:20.036 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:20.036 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:20.036 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:20.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.037 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:20.037 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:20.037 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:20.037 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:20.038 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:20.038 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:20.038 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:20.038 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:20.038 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:20.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:20.039 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:20.039 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667118.715630_513.1", 2020-11-29T16:25:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:18.715630Z", 2020-11-29T16:25:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:20.049 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:20.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.049 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:20.050 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:20.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:20.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:20.052 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:20.091 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:20.092 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:20.330 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:20.367 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:20.368 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:20.368 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:20.368 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:20.368 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:20.369 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:20.369 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667118.715630_513.1", 2020-11-29T16:25:20.369 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:18.715630Z" 2020-11-29T16:25:20.369 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:20.370 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:20.370 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:20.370 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:20.371 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:20.371 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:20.371 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:20.371 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:20.372 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:20.372 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:20.373 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:20.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:20.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:20.374 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:20.415 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:20.787 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:20.787 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:20.788 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:20.788 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:20.788 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:20.788 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:20.788 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:20.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.789 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:20.789 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:20.789 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:20.790 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:20.790 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:20.790 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:20.790 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:20.790 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:20.791 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:20.791 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:20.791 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:20.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:20.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:20.792 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:20.792 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:20.793 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667118.715630_513.1", 2020-11-29T16:25:20.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:20.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:20.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:20.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:18.715630Z", 2020-11-29T16:25:20.794 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:20.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:20.795 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:20.796 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:20.796 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:20.796 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:20.796 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:20.797 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:20.797 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:20.797 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:20.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:20.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:20.798 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:20.799 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:20.799 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:20.799 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:20.799 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:20.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:20.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:20.802 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:20.804 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:21.061 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2020-11-29T16:25:21.062 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2020-11-29T16:25:21.062 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2020-11-29T16:25:21.063 INFO:tasks.rgw_multisite_tests:test reservation cleanup after gateway crash ... ok 2020-11-29T16:25:21.063 INFO:tasks.rgw_multisite_tests:test gateway recovery of persistent notifications ... ok 2020-11-29T16:25:21.064 INFO:tasks.rgw_multisite_tests:test pushing persistent notification via two gateways ... ok 2020-11-29T16:25:21.064 INFO:tasks.rgw_multisite_tests:test pushing persistent notification when one of the endpoints has error ... ok 2020-11-29T16:25:21.065 INFO:tasks.rgw_multisite_tests:test pushing persistent notification http ... ok 2020-11-29T16:25:21.065 INFO:tasks.rgw_multisite_tests:test pushing persistent notification amqp ... ok 2020-11-29T16:25:21.066 INFO:tasks.rgw_multisite_tests:test pushing persistent notification of large notifications ... ok 2020-11-29T16:25:21.067 INFO:tasks.rgw_multisite_tests:test pushing persistent notification pushback ... ok 2020-11-29T16:25:21.067 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-11-29T16:25:21.068 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2020-11-29T16:25:21.068 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2020-11-29T16:25:21.069 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2020-11-29T16:25:21.069 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2020-11-29T16:25:21.070 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2020-11-29T16:25:21.070 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2020-11-29T16:25:21.071 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2020-11-29T16:25:21.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:21.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:21.073 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:21.113 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:21.113 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:21.352 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:21.389 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:21.389 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:21.390 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:21.390 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:21.390 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:21.390 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:21.390 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:21.391 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:21.051408Z" 2020-11-29T16:25:21.391 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:21.391 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:21.391 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:21.392 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:21.392 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:21.392 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:21.392 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:21.392 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:21.393 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:21.393 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:21.394 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:21.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:21.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:21.395 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:21.436 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:21.437 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:21.711 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:21.711 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:21.711 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:21.711 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:21.712 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:21.712 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:21.712 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:21.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:21.712 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:21.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:21.713 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:21.713 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:21.713 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:21.713 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:21.714 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:21.714 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:21.714 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:21.714 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:21.715 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:21.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:21.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:21.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:21.715 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:21.716 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:21.716 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:21.716 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:21.716 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:21.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:21.717 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:21.717 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:21.717 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:21.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:21.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:21.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:21.718 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:21.718 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:21.718 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:21.718 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:21.719 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:21.719 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:21.719 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:21.719 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:21.720 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:21.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:21.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:21.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:21.721 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:21.722 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:21.722 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:21.722 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:21.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:21.723 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:21.723 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:21.723 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:21.723 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:21.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:21.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:21.724 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:21.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:21.725 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:21.725 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:21.725 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:21.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:21.726 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:21.727 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:21.728 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:21.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:21.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:21.729 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:21.771 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:21.771 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:22.031 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:22.032 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:22.032 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:22.032 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:22.032 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:22.033 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:22.033 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:22.033 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:22.033 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:22.033 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:22.034 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:22.034 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:22.034 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:22.034 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:22.035 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:22.035 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:22.035 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:22.035 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:22.035 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:22.036 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:22.036 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:22.036 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:22.036 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:22.036 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:22.037 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:22.037 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:22.037 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:22.037 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:22.037 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:22.038 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:22.038 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:22.038 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:22.038 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:22.039 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:22.039 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:22.039 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:22.039 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:22.039 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:22.040 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:22.040 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:22.040 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:22.041 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:22.041 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:22.042 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:22.042 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:22.042 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:22.042 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:22.043 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:22.043 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:22.043 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:22.043 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:22.043 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:22.044 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:22.044 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:22.044 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:22.044 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:22.045 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:22.045 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:22.046 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:22.046 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:22.046 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:22.047 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:22.047 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:22.047 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:22.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:22.049 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:22.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:22.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:22.050 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:22.094 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:22.094 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:22.366 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:22.366 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:22.367 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:22.367 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:22.367 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:22.367 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:22.368 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:22.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:22.368 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:22.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:22.369 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:22.369 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:22.369 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:22.369 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:22.370 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:22.370 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:22.370 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:22.370 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:22.371 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:22.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:22.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:22.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:22.372 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:22.373 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:22.373 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:22.373 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:22.373 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:22.374 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:22.374 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:22.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:22.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:22.375 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:22.375 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:22.375 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:22.376 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:22.376 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:22.376 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:22.376 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:22.377 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:22.377 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:22.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:22.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:22.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:22.378 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:22.378 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:22.378 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:22.379 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:22.379 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:22.379 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:22.379 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:22.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:22.381 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:22.382 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:22.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:22.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:22.383 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:22.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:22.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:22.385 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:22.426 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:22.426 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:22.663 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:22.697 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:22.698 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:22.698 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:22.698 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:22.699 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:22.699 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:22.699 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:22.700 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:21.051408Z" 2020-11-29T16:25:22.700 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:22.700 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:22.700 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:22.701 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:22.701 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:22.701 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:22.701 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:22.702 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:22.703 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:22.704 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:22.708 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:22.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:22.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:22.710 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:22.750 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:22.751 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:23.009 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:23.010 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:23.010 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:23.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:23.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.022 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:23.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.022 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:23.023 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:23.023 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:23.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.023 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:23.024 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:23.048 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2020-11-29T16:25:23.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:23.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:23.050 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:23.089 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:23.089 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:23.327 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:23.362 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:23.362 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:23.362 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:23.363 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:23.363 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:23.364 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:23.364 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:23.365 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:21.051408Z" 2020-11-29T16:25:23.365 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:23.365 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:23.365 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:23.366 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:23.366 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:23.366 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:23.367 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:23.367 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:23.367 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:23.367 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:23.369 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:23.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:23.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:23.370 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:23.412 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:23.412 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:23.689 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:23.689 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:23.690 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:23.691 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:23.694 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:23.695 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:23.696 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:23.697 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:23.699 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:23.700 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:23.701 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:23.702 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:23.702 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:23.702 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:23.702 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:23.703 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:23.703 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:23.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:23.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:23.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:23.706 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:23.707 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:23.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:23.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:23.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:23.709 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:23.749 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:23.750 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:24.021 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:24.022 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:24.022 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:24.022 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:24.022 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:24.023 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:24.023 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:24.024 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:24.024 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:24.024 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:24.024 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:24.025 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:24.025 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:24.025 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:24.025 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:24.026 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:24.026 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:24.026 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:24.026 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:24.026 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:24.027 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:24.027 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:24.027 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:24.027 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:24.027 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:24.028 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:24.028 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:24.028 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:24.028 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:24.029 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:24.029 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:24.029 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:24.030 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:24.030 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:24.030 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:24.030 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:24.031 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:24.031 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:24.031 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:24.031 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:24.032 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:24.032 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:24.032 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:24.032 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:24.032 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:24.033 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:24.033 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:24.033 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:24.033 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:24.034 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:24.034 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:24.034 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:24.034 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:24.035 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:24.035 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:24.035 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:24.035 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:24.036 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:24.037 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:24.037 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:24.037 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:24.038 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:24.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:24.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:24.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:24.040 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:24.081 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:24.081 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:24.447 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:24.448 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:24.449 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:24.450 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:24.451 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:24.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:24.453 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:24.454 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:24.455 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:24.456 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:24.457 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:24.458 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:24.459 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:24.460 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:24.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:24.462 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:24.463 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:24.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:24.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:24.464 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:24.506 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:24.507 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:24.741 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:24.780 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:24.780 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:24.780 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:24.780 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:24.781 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:24.781 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:24.781 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:24.781 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:21.051408Z" 2020-11-29T16:25:24.782 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:24.782 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:24.782 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:24.782 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:24.782 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:24.783 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:24.783 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:24.783 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:24.783 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:24.784 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:24.784 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:24.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:24.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:24.786 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:24.827 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:24.827 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:25.091 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:25.092 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:25.092 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:25.092 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:25.093 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:25.093 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:25.093 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:25.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.093 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:25.098 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:25.098 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:25.098 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:25.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:25.099 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:25.099 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:25.101 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:25.101 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:25.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.102 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:25.102 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:25.102 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:25.102 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:25.104 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:25.104 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:25.105 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:25.105 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:25.105 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:25.106 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:25.106 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.106 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.106 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:25.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.107 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:25.108 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:25.108 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:25.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.109 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:25.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:25.130 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2020-11-29T16:25:25.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:25.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:25.132 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:25.171 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:25.172 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:25.421 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:25.460 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:25.461 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:25.461 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:25.461 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:25.461 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:25.462 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:25.462 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:25.462 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:21.051408Z" 2020-11-29T16:25:25.462 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:25.462 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:25.463 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:25.463 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:25.463 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:25.463 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:25.463 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:25.464 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:25.464 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:25.464 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:25.465 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:25.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:25.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:25.466 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:25.507 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:25.508 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:25.790 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:25.790 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:25.791 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:25.791 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:25.791 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:25.791 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:25.792 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:25.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.792 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:25.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:25.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:25.793 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:25.793 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:25.793 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:25.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:25.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:25.794 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:25.794 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:25.794 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:25.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:25.795 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:25.795 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:25.796 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:25.796 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:25.796 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:25.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:25.796 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:25.797 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:25.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:25.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:25.798 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:25.798 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:25.798 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:25.798 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:25.799 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:25.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:25.799 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:25.799 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:25.799 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:25.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:25.800 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:25.800 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:25.801 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:25.801 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:25.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:25.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:25.801 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:25.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:25.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:25.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.803 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:25.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:25.803 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:25.803 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:25.803 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:25.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:25.804 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:25.805 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:25.805 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:25.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:25.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:25.807 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:25.849 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:25.850 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:26.120 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:26.121 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:26.121 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:26.121 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:26.121 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:26.122 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:26.122 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:26.122 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:26.123 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:26.123 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:26.123 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:26.123 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:26.124 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:26.124 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:26.124 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:26.124 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:26.125 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:26.125 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:26.125 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:26.126 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:26.126 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:26.126 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:26.126 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:26.127 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:26.127 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:26.127 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:26.127 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:26.128 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:26.128 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:26.128 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:26.128 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:26.129 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:26.129 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:26.129 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:26.130 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:26.130 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:26.130 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:26.130 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:26.131 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:26.131 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:26.131 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:26.131 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:26.132 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:26.132 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:26.132 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:26.132 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:26.133 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:26.133 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:26.133 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:26.133 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:26.134 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:26.134 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:26.134 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:26.134 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:26.135 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:26.135 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:26.135 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:26.135 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:26.136 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:26.136 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:26.136 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:26.136 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:26.137 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:26.137 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:26.138 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:26.139 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:26.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:26.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:26.140 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:26.442 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:26.445 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:26.456 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:26.457 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:26.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:26.465 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:26.465 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:26.469 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:26.470 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:26.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:26.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:26.472 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:26.512 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:26.512 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:26.774 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:26.814 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:26.814 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:26.815 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:26.815 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:26.815 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:26.815 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:26.816 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:26.816 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:21.051408Z" 2020-11-29T16:25:26.816 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:26.816 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:26.816 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:26.817 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:26.817 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:26.817 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:26.817 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:26.818 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:26.818 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:26.818 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:26.820 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:26.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:26.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:26.821 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:26.863 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:26.864 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:27.125 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:27.125 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:27.125 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:27.126 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:27.126 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:27.126 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:27.126 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:27.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.127 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:27.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:27.127 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:27.127 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:27.128 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:27.128 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:27.128 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:27.128 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:27.128 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:27.129 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:27.129 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:27.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.129 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:27.130 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:27.130 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:27.130 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:27.130 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667121.051408_519.1", 2020-11-29T16:25:27.131 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:27.131 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:27.131 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:27.131 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:21.051408Z", 2020-11-29T16:25:27.131 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:27.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.132 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:27.132 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:27.132 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:27.133 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:27.133 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:27.133 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:27.133 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:27.133 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:27.134 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:27.134 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:27.134 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:27.135 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:27.135 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:27.135 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:27.135 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:27.136 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:27.136 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:27.136 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:27.136 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:27.136 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:27.137 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.137 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.137 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:27.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.137 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:27.138 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:27.138 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:27.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.138 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:27.139 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:27.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:27.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:27.197 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:27.233 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:27.234 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:27.470 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:27.506 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:27.507 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:27.507 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:27.507 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:27.508 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:27.508 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:27.508 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667127.187996_523.1", 2020-11-29T16:25:27.508 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:27.187996Z" 2020-11-29T16:25:27.509 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:27.509 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:27.510 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:27.510 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:27.510 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:27.511 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:27.511 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:27.512 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:27.512 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:27.512 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:27.514 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:27.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:27.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:27.516 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:27.558 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:27.559 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:27.825 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667127.187996_523.1", 2020-11-29T16:25:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:27.187996Z", 2020-11-29T16:25:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:27.839 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:27.840 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:27.959 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2020-11-29T16:25:27.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:27.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:27.961 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:28.000 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:28.001 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:28.249 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:28.288 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:28.289 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:28.289 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:28.289 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:28.289 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:28.290 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:28.290 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667127.951233_525.1", 2020-11-29T16:25:28.290 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:27.951233Z" 2020-11-29T16:25:28.290 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:28.291 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:28.291 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:28.291 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:28.291 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:28.291 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:28.292 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:28.292 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:28.292 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:28.293 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:28.294 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:28.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:28.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:28.296 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:28.336 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:28.336 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:28.605 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:28.606 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:28.607 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:28.608 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:28.609 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:28.610 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667127.951233_525.1", 2020-11-29T16:25:28.611 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:27.951233Z", 2020-11-29T16:25:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:28.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:28.613 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:28.614 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:28.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:28.616 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:28.617 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:28.618 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:28.619 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:28.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:28.620 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:28.622 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:28.622 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:28.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:28.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:28.624 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:28.666 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:28.666 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:28.939 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:28.939 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:28.940 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:28.940 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:28.940 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:28.940 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:28.941 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:28.941 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:28.941 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:28.941 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:28.941 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:28.942 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:28.942 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:28.942 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:28.943 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:28.943 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:28.943 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:28.943 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:28.944 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:28.944 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:28.944 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:28.945 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:28.945 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:28.946 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:28.947 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:28.947 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667127.951233_525.1", 2020-11-29T16:25:28.947 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:28.947 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:28.948 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:28.948 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:27.951233Z", 2020-11-29T16:25:28.948 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:28.948 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:28.949 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:28.949 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:28.949 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:28.949 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:28.950 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:28.950 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:28.950 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:28.950 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:28.951 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:28.951 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:28.951 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:28.951 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:28.952 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:28.952 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:28.952 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:28.952 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:28.953 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:28.953 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:28.953 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:28.953 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:28.954 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:28.954 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:28.954 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:28.954 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:28.955 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:28.955 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:28.955 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:28.955 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:28.956 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:28.956 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:28.956 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:28.957 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:28.958 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:28.958 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:28.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:28.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:28.960 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:29.002 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:29.003 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:29.374 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:29.374 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:29.374 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:29.375 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:29.375 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:29.375 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:29.375 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:29.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:29.376 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:29.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:29.376 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:29.376 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:29.377 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:29.377 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:29.377 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:29.377 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:29.377 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:29.378 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:29.378 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:29.378 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:29.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:29.379 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:29.379 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:29.379 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:29.382 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:29.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667127.951233_525.1", 2020-11-29T16:25:29.382 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:29.383 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:29.383 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:29.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:27.951233Z", 2020-11-29T16:25:29.383 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:29.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:29.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:29.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:29.384 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:29.384 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:29.385 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:29.385 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:29.385 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:29.385 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:29.385 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:29.386 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:29.386 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:29.386 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:29.386 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:29.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:29.387 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:29.387 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:29.387 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:29.387 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:29.388 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:29.389 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:29.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:29.390 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:29.390 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:29.390 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:29.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:29.391 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:29.392 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:29.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:29.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:29.393 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:29.460 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:29.460 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:29.707 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:29.743 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:29.744 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:29.745 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:29.745 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:29.746 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:29.746 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:29.747 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667127.951233_525.1", 2020-11-29T16:25:29.748 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:27.951233Z" 2020-11-29T16:25:29.748 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:29.749 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:29.749 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:29.750 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:29.750 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:29.750 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:29.752 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:29.752 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:29.752 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:29.753 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:29.755 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:29.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:29.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:29.757 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:29.802 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:29.802 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:30.056 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:30.056 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:30.057 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:30.057 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:30.057 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:30.058 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:30.058 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:30.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.058 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:30.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:30.059 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:30.059 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:30.059 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:30.060 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:30.060 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:30.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:30.061 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:30.061 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:30.061 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:30.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.062 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:30.062 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:30.063 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:30.063 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:30.063 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667127.951233_525.1", 2020-11-29T16:25:30.063 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:30.063 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:30.064 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:30.064 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:27.951233Z", 2020-11-29T16:25:30.064 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:30.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:30.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:30.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:30.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:30.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:30.067 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:30.067 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:30.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:30.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:30.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:30.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:30.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:30.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:30.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:30.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:30.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:30.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:30.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:30.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:30.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:30.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.072 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:30.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.072 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:30.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:30.073 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:30.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.073 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:30.075 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:30.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:30.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:30.137 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:30.172 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:30.173 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:30.404 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:30.437 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:30.438 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:30.438 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:30.439 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:30.439 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:30.439 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:30.440 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667130.129355_529.1", 2020-11-29T16:25:30.440 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:30.129355Z" 2020-11-29T16:25:30.441 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:30.441 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:30.441 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:30.442 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:30.442 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:30.442 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:30.442 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:30.443 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:30.443 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:30.443 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:30.445 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:30.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:30.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:30.448 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:30.491 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:30.491 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:30.857 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:30.858 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:30.858 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:30.858 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:30.859 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:30.859 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:30.859 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:30.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.860 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:30.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:30.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:30.861 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:30.861 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:30.861 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:30.861 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:30.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:30.862 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:30.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:30.862 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:30.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:30.863 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:30.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:30.864 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:30.864 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667130.129355_529.1", 2020-11-29T16:25:30.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:30.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:30.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:30.865 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:30.129355Z", 2020-11-29T16:25:30.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:30.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:30.866 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:30.866 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:30.866 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:30.867 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:30.867 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:30.867 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:30.867 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:30.867 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:30.868 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:30.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:30.868 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:30.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:30.869 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:30.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:30.869 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:30.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:30.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:30.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:30.870 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:30.871 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:30.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:30.872 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:30.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:30.872 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:30.874 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:31.000 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... ok 2020-11-29T16:25:31.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:31.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:31.002 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:31.040 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:31.040 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:31.269 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:31.305 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:31.305 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:31.305 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:31.306 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:31.306 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:31.306 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:31.306 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667130.989758_531.1", 2020-11-29T16:25:31.307 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:30.989758Z" 2020-11-29T16:25:31.307 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:31.307 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:31.307 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:31.308 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:31.308 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:31.308 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:31.308 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:31.308 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:31.309 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:31.309 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:31.310 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:25:31.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:31.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:31.311 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:31.353 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:31.353 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:31.625 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:31.625 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:31.625 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:31.626 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:31.626 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:31.626 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:31.626 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:31.628 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:31.628 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:31.628 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:31.630 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:31.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:31.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:31.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667130.989758_531.1", 2020-11-29T16:25:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:30.989758Z", 2020-11-29T16:25:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:31.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:31.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:31.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:31.633 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:31.634 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:31.634 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:31.634 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:31.634 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:31.640 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:31.640 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:31.640 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:31.642 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:25:31.643 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:25:31.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:31.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:31.646 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:31.687 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:31.688 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:31.976 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:25:31.977 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:25:31.977 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:25:31.977 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:25:31.977 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:25:31.978 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:31.978 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:31.978 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:31.978 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:25:31.978 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:31.979 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:25:31.979 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:31.979 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:31.979 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:31.980 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:31.980 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:25:31.980 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:31.980 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:31.980 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:31.980 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:31.981 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:31.981 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:31.981 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:25:31.981 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:31.982 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:31.982 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667130.989758_531.1", 2020-11-29T16:25:31.982 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:31.982 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:31.982 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:31.983 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:25:30.989758Z", 2020-11-29T16:25:31.983 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:25:31.983 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:31.983 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:31.983 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:31.984 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:25:31.984 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:31.984 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:31.984 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:31.984 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:31.985 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:31.985 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:31.985 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:31.985 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:31.985 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:31.986 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:31.986 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:31.986 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:25:31.986 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:25:31.986 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:25:31.987 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:31.987 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:25:31.987 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:25:31.987 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:25:31.988 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:25:31.988 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:25:31.988 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:31.988 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:31.988 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:25:31.989 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:31.989 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:25:31.989 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:25:31.989 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:25:31.989 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:31.990 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:25:31.990 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:25:31.991 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:31.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:31.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:31.992 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:32.033 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:32.034 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:32.401 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:32.401 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:32.402 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:32.402 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:32.402 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:32.402 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:32.403 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:32.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:32.403 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:32.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:32.404 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:32.404 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:32.404 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:32.404 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:32.404 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:32.405 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:32.405 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:32.405 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:32.405 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:32.406 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:32.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:32.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:32.406 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:32.406 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:32.407 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:32.407 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667130.989758_531.1", 2020-11-29T16:25:32.407 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:32.407 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:30.989758Z", 2020-11-29T16:25:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:32.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:32.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:32.409 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:32.409 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:32.409 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:32.409 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:32.410 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:32.410 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:32.410 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:32.410 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:32.411 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:32.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:32.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:32.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:32.412 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:32.412 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:32.412 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:32.412 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:32.413 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:32.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:32.414 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:32.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:32.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:32.415 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:32.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:32.415 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:32.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:32.416 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:32.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:32.416 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:32.417 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:32.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:32.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:32.419 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:32.460 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:32.460 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:32.930 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:32.968 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:32.968 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:32.968 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:32.968 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:32.969 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:32.969 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:32.969 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667130.989758_531.1", 2020-11-29T16:25:32.969 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:30.989758Z" 2020-11-29T16:25:32.969 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:32.970 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:32.970 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:32.970 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:32.970 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:32.971 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:32.971 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:32.971 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:32.971 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:32.971 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:32.972 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:32.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:32.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:32.975 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:33.016 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:33.016 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:33.378 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:33.378 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:33.378 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:33.378 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:33.379 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:33.379 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:33.379 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:33.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:33.380 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:33.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:33.380 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:33.380 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:33.381 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:33.381 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:33.381 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:33.381 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:33.382 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:33.382 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:33.382 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:33.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:33.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:33.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:33.383 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:33.383 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:33.384 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:33.384 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667130.989758_531.1", 2020-11-29T16:25:33.384 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:33.384 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:33.384 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:33.385 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:30.989758Z", 2020-11-29T16:25:33.385 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:33.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:33.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:33.386 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:33.386 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:33.386 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:33.386 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:33.386 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:33.387 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:33.387 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:33.387 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:33.387 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:33.387 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:33.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:33.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:33.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:33.388 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:33.388 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:33.389 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:33.389 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:33.389 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:33.389 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:33.389 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:33.390 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:33.390 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:33.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:33.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:33.391 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:33.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:33.391 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:33.391 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:33.391 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:33.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:33.392 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:33.393 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:33.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:33.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:33.449 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:33.484 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:33.485 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:33.714 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:25:33.752 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:25:33.752 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:33.752 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:33.752 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:33.753 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:33.753 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:33.753 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667133.441354_535.1", 2020-11-29T16:25:33.753 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:25:33.441354Z" 2020-11-29T16:25:33.754 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:33.754 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:33.754 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:33.754 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:33.754 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:25:33.755 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:25:33.755 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:25:33.755 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:25:33.755 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:25:33.756 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:25:33.756 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:25:33.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:33.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:25:33.758 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:25:33.800 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:33.800 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:34.170 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:25:34.170 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:25:34.170 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:25:34.171 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:25:34.171 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:25:34.171 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:25:34.171 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:25:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:25:34.172 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:25:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:34.173 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:25:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:25:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:34.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:25:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:34.175 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667133.441354_535.1", 2020-11-29T16:25:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:34.176 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:25:33.441354Z", 2020-11-29T16:25:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:25:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:34.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:25:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:34.178 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:34.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:34.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:34.180 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:25:34.180 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:25:34.180 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:25:34.180 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:25:34.181 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:25:34.181 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:25:34.181 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:25:34.181 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:25:34.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:25:34.182 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:25:34.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:34.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:34.183 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:25:34.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:25:34.183 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:25:34.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:25:34.183 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:25:34.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:25:34.184 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:25:34.185 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:25:34.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:25:34.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:25:34.689 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:25:34.755 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:34.755 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:34.980 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:34.974+0000 7fdd8d010080 1 waiting for incremental sync to catch up: 2020-11-29T16:25:34.980 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.107.3, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000001.126.5] 2020-11-29T16:25:34.980 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.107.3, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:25:39.984 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:39.978+0000 7fdd8d010080 1 waiting for incremental sync to catch up: 2020-11-29T16:25:39.984 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.107.3, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000001.126.5] 2020-11-29T16:25:39.985 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.107.3, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:25:44.988 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:44.982+0000 7fdd8d010080 1 waiting for incremental sync to catch up: 2020-11-29T16:25:44.988 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.107.3, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000001.126.5] 2020-11-29T16:25:44.988 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.107.3, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:25:49.992 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:49.986+0000 7fdd8d010080 1 bucket sync caught up with source: 2020-11-29T16:25:49.992 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.107.3, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:25:49.992 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.107.3, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:25:49.993 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:49.986+0000 7fdd8d010080 0 bucket checkpoint complete 2020-11-29T16:25:50.235 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "2" type: "OBJECT_CREATE" 2020-11-29T16:25:50.236 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "3" type: "OBJECT_CREATE" 2020-11-29T16:25:50.236 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "0" type: "OBJECT_CREATE" 2020-11-29T16:25:50.236 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "1" type: "OBJECT_CREATE" 2020-11-29T16:25:50.237 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "4" type: "OBJECT_CREATE" 2020-11-29T16:25:50.237 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "6" type: "OBJECT_CREATE" 2020-11-29T16:25:50.238 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "5" type: "OBJECT_CREATE" 2020-11-29T16:25:50.238 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-11-29T16:25:50.238 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "8" type: "OBJECT_CREATE" 2020-11-29T16:25:50.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "7" type: "OBJECT_CREATE" 2020-11-29T16:25:50.239 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "9" type: "OBJECT_CREATE" 2020-11-29T16:25:50.249 DEBUG:tasks.rgw_multi.tests_ps:superfluous events are found 2020-11-29T16:25:50.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:25:50.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-81', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:25:50.344 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-81 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:25:50.394 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:25:50.395 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:25:50.655 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:50.646+0000 7f1476293080 1 waiting for incremental sync to catch up: 2020-11-29T16:25:50.655 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000005.113.5, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:25:50.655 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000005.113.5, , 00000000002.73.5, 00000000002.124.5, 00000000002.109.5, , 00000000002.79.5, 00000000002.113.5, 00000000002.157.5, 00000000004.132.5] 2020-11-29T16:25:55.660 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:25:55.654+0000 7f1476293080 1 waiting for incremental sync to catch up: 2020-11-29T16:25:55.660 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000005.113.5, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:25:55.660 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000005.113.5, , 00000000002.73.5, 00000000002.124.5, 00000000002.109.5, , 00000000002.79.5, 00000000002.113.5, 00000000002.157.5, 00000000004.132.5] 2020-11-29T16:26:00.664 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:00.658+0000 7f1476293080 1 waiting for incremental sync to catch up: 2020-11-29T16:26:00.664 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000005.113.5, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:26:00.665 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000005.113.5, , 00000000002.73.5, 00000000002.124.5, 00000000002.109.5, , 00000000002.79.5, 00000000002.113.5, 00000000002.157.5, 00000000004.132.5] 2020-11-29T16:26:05.669 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:05.662+0000 7f1476293080 1 waiting for incremental sync to catch up: 2020-11-29T16:26:05.670 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000005.113.5, , 00000000001.71.5, 00000000001.122.5, 00000000001.102.5, , 00000000001.77.5, 00000000001.111.5, 00000000001.155.5, 00000000002.128.5] 2020-11-29T16:26:05.670 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000005.113.5, , 00000000002.73.5, 00000000002.124.5, 00000000002.109.5, , 00000000002.79.5, 00000000002.113.5, 00000000002.157.5, 00000000004.132.5] 2020-11-29T16:26:10.673 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:10.666+0000 7f1476293080 1 bucket sync caught up with source: 2020-11-29T16:26:10.674 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000005.113.5, , 00000000002.73.5, 00000000002.124.5, 00000000002.109.5, , 00000000002.79.5, 00000000002.113.5, 00000000002.157.5, 00000000004.132.5] 2020-11-29T16:26:10.674 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000005.113.5, , 00000000002.73.5, 00000000002.124.5, 00000000002.109.5, , 00000000002.79.5, 00000000002.113.5, 00000000002.157.5, 00000000004.132.5] 2020-11-29T16:26:10.675 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:10.666+0000 7f1476293080 0 bucket checkpoint complete 2020-11-29T16:26:10.907 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... ok 2020-11-29T16:26:10.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:10.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:10.910 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:10.956 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:10.957 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:11.206 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:26:11.244 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:26:11.244 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:11.244 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:11.245 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:11.245 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:11.245 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:11.245 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667170.897189_538.1", 2020-11-29T16:26:11.245 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:26:10.897189Z" 2020-11-29T16:26:11.246 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:11.246 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:11.246 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:11.246 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:11.247 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:11.247 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:11.247 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:11.247 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:11.247 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:11.248 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:26:11.249 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:26:11.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:11.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:11.250 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:11.292 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:11.292 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:11.566 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:11.566 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:26:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:26:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:26:11.567 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:26:11.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:26:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:11.569 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:26:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:11.570 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:11.571 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:26:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667170.897189_538.1", 2020-11-29T16:26:11.572 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:10.897189Z", 2020-11-29T16:26:11.573 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:26:11.574 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:11.575 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:11.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:26:11.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:11.578 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:11.579 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:11.580 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:26:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:26:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:26:11.581 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:11.581 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:11.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:26:11.583 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:26:11.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:11.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:11.584 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:11.626 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:11.626 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:11.888 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:26:11.888 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:26:11.888 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:26:11.889 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:26:11.889 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:26:11.889 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:11.889 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:26:11.890 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:11.890 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:26:11.891 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:11.891 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:26:11.891 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:11.891 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:11.892 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:11.892 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:11.892 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:26:11.893 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:11.893 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:11.893 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:11.893 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:11.894 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:11.894 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:11.894 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:26:11.894 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:11.895 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:11.895 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667170.897189_538.1", 2020-11-29T16:26:11.895 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:11.895 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:11.895 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:11.896 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:26:10.897189Z", 2020-11-29T16:26:11.896 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:26:11.896 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:11.896 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:11.897 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:11.897 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:26:11.897 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:11.897 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:11.898 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:11.898 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:11.898 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:11.898 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:11.898 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:26:11.899 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:11.899 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:11.899 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:11.899 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:11.900 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:26:11.900 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:11.900 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:11.900 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:11.900 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:11.901 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:11.901 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:11.901 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:26:11.901 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:11.902 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:11.902 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:11.902 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:26:11.903 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:11.903 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:26:11.903 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:26:11.903 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:26:11.903 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:11.904 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:26:11.905 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:26:11.905 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:26:11.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:11.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:11.907 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:11.949 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:11.949 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:12.315 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:26:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:26:12.316 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:26:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:12.317 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:26:12.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:26:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:12.319 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:26:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:12.320 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:26:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:12.322 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667170.897189_538.1", 2020-11-29T16:26:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:12.323 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:10.897189Z", 2020-11-29T16:26:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:26:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:12.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:12.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:26:12.328 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:12.329 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:12.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.331 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:26:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:26:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:26:12.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.332 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:12.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:26:12.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:12.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:12.336 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:12.369 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:12.370 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:12.618 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:26:12.649 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:26:12.650 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:12.650 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:12.650 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:12.651 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:12.651 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:12.651 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667170.897189_538.1", 2020-11-29T16:26:12.651 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:26:10.897189Z" 2020-11-29T16:26:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:12.652 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:12.653 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:12.654 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:12.654 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:26:12.655 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:26:12.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:12.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:12.657 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:12.699 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:12.699 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:12.968 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:12.969 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:26:12.969 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:12.970 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:26:12.970 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:26:12.971 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:12.971 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:12.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.972 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:26:12.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:12.976 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:26:12.976 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:12.976 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:12.977 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:12.977 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:12.978 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:26:12.978 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:12.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:12.979 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:12.979 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.980 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:12.981 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:26:12.981 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:12.981 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:12.982 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667170.897189_538.1", 2020-11-29T16:26:12.982 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:12.983 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:12.983 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:12.983 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:10.897189Z", 2020-11-29T16:26:12.984 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:12.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:12.986 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:26:12.986 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:12.986 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:12.987 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:12.987 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:12.988 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:12.988 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:12.988 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:12.989 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:12.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:12.990 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:26:12.991 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:12.991 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:12.992 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:12.992 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:12.992 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:12.993 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:12.993 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:12.994 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:12.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.995 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:12.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:12.996 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:26:12.996 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:26:12.996 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:26:12.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:12.997 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:12.999 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:26:13.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:13.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:13.014 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic list --uid rgw-multisite-test-user --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:26:13.080 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:13.080 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:13.459 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: "topics": [ 2020-11-29T16:26:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: "topic": { 2020-11-29T16:26:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: "user": "rgw-multisite-test-user", 2020-11-29T16:26:13.460 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82_topic", 2020-11-29T16:26:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: "dest": { 2020-11-29T16:26:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_name": "", 2020-11-29T16:26:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: "oid_prefix": "", 2020-11-29T16:26:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint": "", 2020-11-29T16:26:13.461 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint_args": "", 2020-11-29T16:26:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint_topic": "kcmyep-82_topic", 2020-11-29T16:26:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: "stored_secret": "false", 2020-11-29T16:26:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: "persistent": "false" 2020-11-29T16:26:13.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: "arn": "arn:aws:sns:test-zonegroup::kcmyep-82_topic", 2020-11-29T16:26:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: "opaqueData": "" 2020-11-29T16:26:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: "subs": [] 2020-11-29T16:26:13.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:13.464 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:13.464 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:13.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'kcmyep-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:13.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'kcmyep-82_topic', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:13.466 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 topic get --uid rgw-multisite-test-user --topic kcmyep-82_topic --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:26:13.548 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:13.548 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:13.921 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:13.921 INFO:teuthology.orchestra.run.smithi184.stdout: "topic": { 2020-11-29T16:26:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "user": "rgw-multisite-test-user", 2020-11-29T16:26:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82_topic", 2020-11-29T16:26:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "dest": { 2020-11-29T16:26:13.922 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_name": "", 2020-11-29T16:26:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "oid_prefix": "", 2020-11-29T16:26:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint": "", 2020-11-29T16:26:13.923 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint_args": "", 2020-11-29T16:26:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint_topic": "kcmyep-82_topic", 2020-11-29T16:26:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "stored_secret": "false", 2020-11-29T16:26:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: "persistent": "false" 2020-11-29T16:26:13.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "arn": "arn:aws:sns:test-zonegroup::kcmyep-82_topic", 2020-11-29T16:26:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "opaqueData": "" 2020-11-29T16:26:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:13.925 INFO:teuthology.orchestra.run.smithi184.stdout: "subs": [] 2020-11-29T16:26:13.926 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:13.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:13.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'list', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:13.948 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic list --uid rgw-multisite-test-user --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:26:13.986 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:13.986 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:14.270 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:26:14.270 INFO:teuthology.orchestra.run.smithi168.stdout: "topics": [ 2020-11-29T16:26:14.271 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:14.271 INFO:teuthology.orchestra.run.smithi168.stdout: "topic": { 2020-11-29T16:26:14.271 INFO:teuthology.orchestra.run.smithi168.stdout: "user": "rgw-multisite-test-user", 2020-11-29T16:26:14.271 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-82_topic", 2020-11-29T16:26:14.271 INFO:teuthology.orchestra.run.smithi168.stdout: "dest": { 2020-11-29T16:26:14.272 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_name": "", 2020-11-29T16:26:14.272 INFO:teuthology.orchestra.run.smithi168.stdout: "oid_prefix": "", 2020-11-29T16:26:14.272 INFO:teuthology.orchestra.run.smithi168.stdout: "push_endpoint": "amqp://127.0.0.1:7001/vhost_1", 2020-11-29T16:26:14.272 INFO:teuthology.orchestra.run.smithi168.stdout: "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1", 2020-11-29T16:26:14.273 INFO:teuthology.orchestra.run.smithi168.stdout: "push_endpoint_topic": "kcmyep-82_topic", 2020-11-29T16:26:14.273 INFO:teuthology.orchestra.run.smithi168.stdout: "stored_secret": "false", 2020-11-29T16:26:14.273 INFO:teuthology.orchestra.run.smithi168.stdout: "persistent": "false" 2020-11-29T16:26:14.273 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:14.273 INFO:teuthology.orchestra.run.smithi168.stdout: "arn": "arn:aws:sns:test-zonegroup::kcmyep-82_topic", 2020-11-29T16:26:14.274 INFO:teuthology.orchestra.run.smithi168.stdout: "opaqueData": "" 2020-11-29T16:26:14.274 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:14.274 INFO:teuthology.orchestra.run.smithi168.stdout: "subs": [] 2020-11-29T16:26:14.274 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:14.274 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:26:14.275 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:26:14.276 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'kcmyep-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:14.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'topic', 'get', '--uid', 'rgw-multisite-test-user', '--topic', 'kcmyep-82_topic', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:14.277 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 topic get --uid rgw-multisite-test-user --topic kcmyep-82_topic --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-11-29T16:26:14.358 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:14.359 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:14.642 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:26:14.643 INFO:teuthology.orchestra.run.smithi168.stdout: "topic": { 2020-11-29T16:26:14.643 INFO:teuthology.orchestra.run.smithi168.stdout: "user": "rgw-multisite-test-user", 2020-11-29T16:26:14.643 INFO:teuthology.orchestra.run.smithi168.stdout: "name": "kcmyep-82_topic", 2020-11-29T16:26:14.644 INFO:teuthology.orchestra.run.smithi168.stdout: "dest": { 2020-11-29T16:26:14.644 INFO:teuthology.orchestra.run.smithi168.stdout: "bucket_name": "", 2020-11-29T16:26:14.644 INFO:teuthology.orchestra.run.smithi168.stdout: "oid_prefix": "", 2020-11-29T16:26:14.644 INFO:teuthology.orchestra.run.smithi168.stdout: "push_endpoint": "amqp://127.0.0.1:7001/vhost_1", 2020-11-29T16:26:14.644 INFO:teuthology.orchestra.run.smithi168.stdout: "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value=amqp://127.0.0.1:7001/vhost_1&Attributes.entry.2.key=amqp-exchange&Attributes.entry.2.value=amqp.direct&Attributes.entry.3.key=amqp-ack-level&Attributes.entry.3.value=none&Version=2010-03-31&amqp-ack-level=none&amqp-exchange=amqp.direct&push-endpoint=amqp://127.0.0.1:7001/vhost_1", 2020-11-29T16:26:14.645 INFO:teuthology.orchestra.run.smithi168.stdout: "push_endpoint_topic": "kcmyep-82_topic", 2020-11-29T16:26:14.645 INFO:teuthology.orchestra.run.smithi168.stdout: "stored_secret": "false", 2020-11-29T16:26:14.645 INFO:teuthology.orchestra.run.smithi168.stdout: "persistent": "false" 2020-11-29T16:26:14.645 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:14.645 INFO:teuthology.orchestra.run.smithi168.stdout: "arn": "arn:aws:sns:test-zonegroup::kcmyep-82_topic", 2020-11-29T16:26:14.646 INFO:teuthology.orchestra.run.smithi168.stdout: "opaqueData": "" 2020-11-29T16:26:14.646 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:14.646 INFO:teuthology.orchestra.run.smithi168.stdout: "subs": [] 2020-11-29T16:26:14.647 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:26:14.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:14.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:14.692 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:14.728 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:14.728 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:14.967 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:26:15.005 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:26:15.005 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:15.005 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:15.006 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:15.006 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:15.006 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:15.006 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667174.684079_542.1", 2020-11-29T16:26:15.007 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:26:14.684079Z" 2020-11-29T16:26:15.007 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:15.007 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:15.007 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:15.007 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:15.008 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:15.008 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:15.008 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:15.008 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:15.009 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:15.009 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:26:15.010 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:26:15.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:15.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:15.012 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:15.052 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:15.053 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:15.313 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:26:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:26:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:26:15.315 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:15.315 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:15.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:15.315 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:26:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:26:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:15.316 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:26:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:15.317 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:26:15.318 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667174.684079_542.1", 2020-11-29T16:26:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:15.319 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:14.684079Z", 2020-11-29T16:26:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:15.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:26:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:15.321 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:15.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:15.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:15.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:15.322 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:26:15.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:15.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:15.325 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:15.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:15.326 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:26:15.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:26:15.326 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:26:15.326 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:15.327 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:15.328 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:26:15.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'kcmyep-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:15.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'get', '--uid', 'rgw-multisite-test-user', '--subscription', 'kcmyep-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:15.354 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription get --uid rgw-multisite-test-user --subscription kcmyep-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:26:15.410 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:15.411 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:15.681 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:15.682 INFO:teuthology.orchestra.run.smithi184.stdout: "user": "rgw-multisite-test-user", 2020-11-29T16:26:15.682 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82_sub", 2020-11-29T16:26:15.682 INFO:teuthology.orchestra.run.smithi184.stdout: "topic": "kcmyep-82_topic", 2020-11-29T16:26:15.683 INFO:teuthology.orchestra.run.smithi184.stdout: "dest": { 2020-11-29T16:26:15.683 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_name": "pubsub-rgw-multisite-test-user-kcmyep-82_topic", 2020-11-29T16:26:15.683 INFO:teuthology.orchestra.run.smithi184.stdout: "oid_prefix": "kcmyep-82_sub/", 2020-11-29T16:26:15.683 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint": "", 2020-11-29T16:26:15.683 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint_args": "topic=kcmyep-82_topic", 2020-11-29T16:26:15.684 INFO:teuthology.orchestra.run.smithi184.stdout: "push_endpoint_topic": "kcmyep-82_topic", 2020-11-29T16:26:15.684 INFO:teuthology.orchestra.run.smithi184.stdout: "stored_secret": "false", 2020-11-29T16:26:15.684 INFO:teuthology.orchestra.run.smithi184.stdout: "persistent": "false" 2020-11-29T16:26:15.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:15.685 INFO:teuthology.orchestra.run.smithi184.stdout: "s3_id": "" 2020-11-29T16:26:15.685 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:20.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:26:20.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-82', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:26:20.580 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-82 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:26:20.628 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:20.628 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:20.892 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:20.886+0000 7f3ab8f7e080 1 waiting for incremental sync to catch up: 2020-11-29T16:26:20.892 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.105.5, 00000000004.75.3, 00000000001.76.5, 00000000001.92.5, 00000000001.88.5, 00000000001.81.5, 00000000001.93.5, 00000000001.83.5, 00000000001.116.5, 00000000001.65.5, 00000000001.94.5] 2020-11-29T16:26:20.892 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000011.125.5, 00000000013.93.5, 00000000010.94.5, 00000000010.110.5, 00000000009.104.5, 00000000007.93.5, 00000000010.121.5, 00000000010.111.5, 00000000011.136.5, 00000000011.85.5, 00000000010.112.5] 2020-11-29T16:26:25.896 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:25.890+0000 7f3ab8f7e080 1 waiting for incremental sync to catch up: 2020-11-29T16:26:25.896 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.105.5, 00000000004.75.3, 00000000001.76.5, 00000000001.92.5, 00000000001.88.5, 00000000001.81.5, 00000000001.93.5, 00000000001.83.5, 00000000001.116.5, 00000000001.65.5, 00000000001.94.5] 2020-11-29T16:26:25.896 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000011.125.5, 00000000013.93.5, 00000000010.94.5, 00000000010.110.5, 00000000009.104.5, 00000000007.93.5, 00000000010.121.5, 00000000010.111.5, 00000000011.136.5, 00000000011.85.5, 00000000010.112.5] 2020-11-29T16:26:30.901 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:30.894+0000 7f3ab8f7e080 1 bucket sync caught up with source: 2020-11-29T16:26:30.901 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000011.125.5, 00000000013.93.5, 00000000010.94.5, 00000000010.110.5, 00000000009.104.5, 00000000007.93.5, 00000000010.121.5, 00000000010.111.5, 00000000011.136.5, 00000000011.85.5, 00000000010.112.5] 2020-11-29T16:26:30.901 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000011.125.5, 00000000013.93.5, 00000000010.94.5, 00000000010.110.5, 00000000009.104.5, 00000000007.93.5, 00000000010.121.5, 00000000010.111.5, 00000000011.136.5, 00000000011.85.5, 00000000010.112.5] 2020-11-29T16:26:30.901 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:30.894+0000 7f3ab8f7e080 0 bucket checkpoint complete 2020-11-29T16:26:31.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'kcmyep-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:31.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'kcmyep-82_sub', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:31.009 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription kcmyep-82_sub --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:26:31.049 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:31.050 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:31.316 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:31.317 INFO:teuthology.orchestra.run.smithi184.stdout: "next_marker": "kcmyep-82_sub/1606667190.432672.d351de38", 2020-11-29T16:26:31.317 INFO:teuthology.orchestra.run.smithi184.stdout: "is_truncated": "true", 2020-11-29T16:26:31.317 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2020-11-29T16:26:31.317 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.317 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667175.834312.a6645f2c", 2020-11-29T16:26:31.318 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.318 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:15.834312Z", 2020-11-29T16:26:31.318 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.318 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.319 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:15.750966Z" 2020-11-29T16:26:31.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.319 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.319 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.319 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.320 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.320 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.320 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.320 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "2" 2020-11-29T16:26:31.321 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.321 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.321 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667175.883992.4caae6af", 2020-11-29T16:26:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:15.883992Z", 2020-11-29T16:26:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:15.706056Z" 2020-11-29T16:26:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "1" 2020-11-29T16:26:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667175.921348.5f96e32e", 2020-11-29T16:26:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:15.921348Z", 2020-11-29T16:26:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:15.687873Z" 2020-11-29T16:26:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "0" 2020-11-29T16:26:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.027056.34d56745", 2020-11-29T16:26:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.027056Z", 2020-11-29T16:26:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:15.882959Z" 2020-11-29T16:26:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "5" 2020-11-29T16:26:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.069031.d06d567e", 2020-11-29T16:26:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.069031Z", 2020-11-29T16:26:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.336 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.336 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:15.926638Z" 2020-11-29T16:26:31.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.337 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.337 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.337 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.337 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.338 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.338 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.338 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "6" 2020-11-29T16:26:31.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.339 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.075076.2187e39d", 2020-11-29T16:26:31.339 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.340 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.075076Z", 2020-11-29T16:26:31.340 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.340 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.340 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.015580Z" 2020-11-29T16:26:31.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "8" 2020-11-29T16:26:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.113546.43dbe47b", 2020-11-29T16:26:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.113546Z", 2020-11-29T16:26:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:15.794750Z" 2020-11-29T16:26:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "3" 2020-11-29T16:26:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.117500.588c4458", 2020-11-29T16:26:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.117500Z", 2020-11-29T16:26:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.350 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.350 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:15.838526Z" 2020-11-29T16:26:31.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.351 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.351 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.351 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.352 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.353 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.353 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "4" 2020-11-29T16:26:31.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.354 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.129483.2187e39d", 2020-11-29T16:26:31.355 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.129483Z", 2020-11-29T16:26:31.355 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.355 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.356 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.015580Z" 2020-11-29T16:26:31.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.356 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.356 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.356 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.357 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.357 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.358 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.358 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "8" 2020-11-29T16:26:31.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.359 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.197504.2187e39d", 2020-11-29T16:26:31.359 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.359 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.197504Z", 2020-11-29T16:26:31.360 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.360 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.360 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.015580Z" 2020-11-29T16:26:31.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.360 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.361 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.361 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.361 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.362 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.362 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.362 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "8" 2020-11-29T16:26:31.362 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.362 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.363 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.270166.ece68696", 2020-11-29T16:26:31.363 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.363 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.270166Z", 2020-11-29T16:26:31.364 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.364 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.364 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.120382Z" 2020-11-29T16:26:31.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.364 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.365 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.365 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.365 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.366 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.366 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.366 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "10" 2020-11-29T16:26:31.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.367 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.309444.f8c5905b", 2020-11-29T16:26:31.367 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.368 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.309444Z", 2020-11-29T16:26:31.368 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.368 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.369 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:15.970983Z" 2020-11-29T16:26:31.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.369 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.369 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.370 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.370 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.370 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.371 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.371 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "7" 2020-11-29T16:26:31.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.372 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667176.874774.951e61d7", 2020-11-29T16:26:31.372 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.372 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:16.874774Z", 2020-11-29T16:26:31.372 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.373 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.373 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.738720Z" 2020-11-29T16:26:31.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.374 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.374 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.374 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.374 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.375 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.375 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.375 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "24" 2020-11-29T16:26:31.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.376 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667177.678829.5d8f9078", 2020-11-29T16:26:31.377 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.377 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:17.678829Z", 2020-11-29T16:26:31.377 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.377 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.377 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.446922Z" 2020-11-29T16:26:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.378 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "40" 2020-11-29T16:26:31.379 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.279388.a095a552", 2020-11-29T16:26:31.380 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.279388Z", 2020-11-29T16:26:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.066077Z" 2020-11-29T16:26:31.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.382 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.383 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.383 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.383 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "9" 2020-11-29T16:26:31.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.384 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.384 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.279582.54e31a53", 2020-11-29T16:26:31.384 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.385 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.279582Z", 2020-11-29T16:26:31.385 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.385 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.385 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.475244Z" 2020-11-29T16:26:31.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.386 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.386 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.424 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.424 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.424 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.425 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.425 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "18" 2020-11-29T16:26:31.425 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.426 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.426 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.280188.83dacbc5", 2020-11-29T16:26:31.427 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.427 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.280188Z", 2020-11-29T16:26:31.427 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.428 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.428 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.090986Z" 2020-11-29T16:26:31.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.429 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.429 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.429 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.429 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.430 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.430 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.430 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "32" 2020-11-29T16:26:31.430 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.431 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.431 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.432 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.281163.dae0d98f", 2020-11-29T16:26:31.432 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.432 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.281163Z", 2020-11-29T16:26:31.433 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.433 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.433 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.606858Z" 2020-11-29T16:26:31.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.434 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.435 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.435 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.435 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "21" 2020-11-29T16:26:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.437 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.282576.f1278cf4", 2020-11-29T16:26:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.282576Z", 2020-11-29T16:26:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.438 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.067055Z" 2020-11-29T16:26:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.439 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.440 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.441 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.442 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "54" 2020-11-29T16:26:31.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.283345.11df8ab9", 2020-11-29T16:26:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.444 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.283345Z", 2020-11-29T16:26:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.445 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.731228Z" 2020-11-29T16:26:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.446 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.448 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "69" 2020-11-29T16:26:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.286654.ce7aeaf1", 2020-11-29T16:26:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.286654Z", 2020-11-29T16:26:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.451 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.311045Z" 2020-11-29T16:26:31.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.452 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.452 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.453 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.453 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.454 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.454 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.455 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "104" 2020-11-29T16:26:31.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.456 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.456 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.286884.349653f3", 2020-11-29T16:26:31.456 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.457 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.286884Z", 2020-11-29T16:26:31.457 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.457 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.458 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.151142Z" 2020-11-29T16:26:31.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.458 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.459 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.459 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.460 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.460 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.460 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.461 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "78" 2020-11-29T16:26:31.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.463 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.286981.2dc0b9a5", 2020-11-29T16:26:31.463 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.463 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.286981Z", 2020-11-29T16:26:31.463 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.464 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.464 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.111147Z" 2020-11-29T16:26:31.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.465 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.465 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.465 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.465 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.466 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.467 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.467 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "55" 2020-11-29T16:26:31.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.468 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.287251.b1459312", 2020-11-29T16:26:31.469 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.469 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.287251Z", 2020-11-29T16:26:31.469 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.469 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.470 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.199743Z" 2020-11-29T16:26:31.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.470 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.471 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.471 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.471 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.472 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.472 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.472 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "57" 2020-11-29T16:26:31.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.474 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.288049.76ae73a4", 2020-11-29T16:26:31.474 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.474 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.288049Z", 2020-11-29T16:26:31.474 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.475 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.475 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.266986Z" 2020-11-29T16:26:31.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.475 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.476 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.476 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.476 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "36" 2020-11-29T16:26:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.479 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.288148.1ff2ec1e", 2020-11-29T16:26:31.479 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.479 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.288148Z", 2020-11-29T16:26:31.480 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.480 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.480 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.922723Z" 2020-11-29T16:26:31.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.481 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.481 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.481 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.481 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.482 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.482 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.483 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "73" 2020-11-29T16:26:31.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.484 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.288243.8d1b1b90", 2020-11-29T16:26:31.485 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.485 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.288243Z", 2020-11-29T16:26:31.485 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.485 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.486 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.462722Z" 2020-11-29T16:26:31.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.486 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.487 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.487 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.487 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.488 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.488 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.488 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "63" 2020-11-29T16:26:31.488 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.489 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.289076.d833d56d", 2020-11-29T16:26:31.489 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.490 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.289076Z", 2020-11-29T16:26:31.490 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.490 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.490 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.914342Z" 2020-11-29T16:26:31.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.491 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.491 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.491 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.492 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.492 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.492 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.493 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "28" 2020-11-29T16:26:31.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.493 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.494 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.289318.e23f8511", 2020-11-29T16:26:31.494 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.494 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.289318Z", 2020-11-29T16:26:31.494 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.495 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.495 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.175124Z" 2020-11-29T16:26:31.495 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.495 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.496 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.496 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.496 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.496 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.497 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.497 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "101" 2020-11-29T16:26:31.497 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.497 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.498 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.289592.68b6146f", 2020-11-29T16:26:31.498 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.498 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.289592Z", 2020-11-29T16:26:31.499 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.499 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.499 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.598859Z" 2020-11-29T16:26:31.499 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.499 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.500 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.500 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.500 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.501 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.501 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.501 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "88" 2020-11-29T16:26:31.501 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.501 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.502 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.289889.aa8e337c", 2020-11-29T16:26:31.502 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.502 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.289889Z", 2020-11-29T16:26:31.503 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.503 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.503 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.403129Z" 2020-11-29T16:26:31.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.504 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.504 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.504 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.504 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "106" 2020-11-29T16:26:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.290183.06600672", 2020-11-29T16:26:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.506 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.290183Z", 2020-11-29T16:26:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.774778Z" 2020-11-29T16:26:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.507 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.508 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "92" 2020-11-29T16:26:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.292826.b0241c1c", 2020-11-29T16:26:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.510 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.292826Z", 2020-11-29T16:26:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.731123Z" 2020-11-29T16:26:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.511 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "91" 2020-11-29T16:26:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.292913.bf940825", 2020-11-29T16:26:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.514 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.292913Z", 2020-11-29T16:26:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.107555Z" 2020-11-29T16:26:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.515 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.516 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.517 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.517 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "77" 2020-11-29T16:26:31.517 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.517 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.518 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.518 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.293027.21f319b4", 2020-11-29T16:26:31.518 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.518 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.293027Z", 2020-11-29T16:26:31.519 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.519 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.519 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.219378Z" 2020-11-29T16:26:31.519 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.520 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.520 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.520 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.520 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.520 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.521 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.521 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.521 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "102" 2020-11-29T16:26:31.521 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.522 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.522 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.522 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.522 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.293114.dae2eef4", 2020-11-29T16:26:31.522 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.523 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.293114Z", 2020-11-29T16:26:31.523 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.523 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.523 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.287685Z" 2020-11-29T16:26:31.523 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.524 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.524 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.524 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.524 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.525 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.525 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.525 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.525 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "59" 2020-11-29T16:26:31.525 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.526 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.526 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.293870.d214d0b8", 2020-11-29T16:26:31.527 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.527 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.293870Z", 2020-11-29T16:26:31.527 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.527 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.527 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.402884Z" 2020-11-29T16:26:31.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.528 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.528 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.528 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.529 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.529 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.529 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.529 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "39" 2020-11-29T16:26:31.530 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.530 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.530 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.531 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.293931.15187aba", 2020-11-29T16:26:31.531 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.531 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.293931Z", 2020-11-29T16:26:31.531 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.531 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.532 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.535048Z" 2020-11-29T16:26:31.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.532 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.532 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.533 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.533 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.533 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.533 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.533 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.534 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "42" 2020-11-29T16:26:31.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.534 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.535 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.294163.4a427686", 2020-11-29T16:26:31.535 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.535 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.294163Z", 2020-11-29T16:26:31.535 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.536 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.536 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.890852Z" 2020-11-29T16:26:31.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.536 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.536 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.537 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.537 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.537 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.538 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.538 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "50" 2020-11-29T16:26:31.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.539 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.294323.16bd062d", 2020-11-29T16:26:31.539 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.539 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.294323Z", 2020-11-29T16:26:31.539 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.540 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.540 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.911004Z" 2020-11-29T16:26:31.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.540 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.541 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.541 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.541 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.542 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.542 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.542 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "95" 2020-11-29T16:26:31.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.543 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.294503.f71cd4ae", 2020-11-29T16:26:31.543 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.544 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.294503Z", 2020-11-29T16:26:31.544 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.544 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.544 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.956110Z" 2020-11-29T16:26:31.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.545 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.545 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.545 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.545 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.546 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.546 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "96" 2020-11-29T16:26:31.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.547 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.547 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.434852.c634653b", 2020-11-29T16:26:31.547 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.434852Z", 2020-11-29T16:26:31.548 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.548 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.548 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.802804Z" 2020-11-29T16:26:31.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.549 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.549 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.549 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.549 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.550 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.550 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.550 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "48" 2020-11-29T16:26:31.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.551 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.551 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.435403.9d6529c3", 2020-11-29T16:26:31.551 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.552 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.435403Z", 2020-11-29T16:26:31.552 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.552 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.552 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.243324Z" 2020-11-29T16:26:31.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.553 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.553 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.553 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.553 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.554 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.554 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.554 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "80" 2020-11-29T16:26:31.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.555 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.435614.5fec4685", 2020-11-29T16:26:31.555 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.556 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.435614Z", 2020-11-29T16:26:31.556 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.556 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.556 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.779063Z" 2020-11-29T16:26:31.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.557 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.557 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.557 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.557 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.558 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.558 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.558 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.558 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "70" 2020-11-29T16:26:31.558 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.559 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.559 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.436404.1a3c09bc", 2020-11-29T16:26:31.559 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.560 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.436404Z", 2020-11-29T16:26:31.560 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.560 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.560 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.167323Z" 2020-11-29T16:26:31.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.561 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.561 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.561 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.561 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.562 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.562 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.562 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.562 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "11" 2020-11-29T16:26:31.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.563 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.563 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.436515.388f0edd", 2020-11-29T16:26:31.564 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.564 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.436515Z", 2020-11-29T16:26:31.564 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.564 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.565 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.638648Z" 2020-11-29T16:26:31.565 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.565 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.565 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.565 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "67" 2020-11-29T16:26:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.436637.e60ebcbe", 2020-11-29T16:26:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.436637Z", 2020-11-29T16:26:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.288589Z" 2020-11-29T16:26:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "81" 2020-11-29T16:26:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.437254.a0be60ae", 2020-11-29T16:26:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.437254Z", 2020-11-29T16:26:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.846995Z" 2020-11-29T16:26:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "49" 2020-11-29T16:26:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.437328.98dad10d", 2020-11-29T16:26:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.437328Z", 2020-11-29T16:26:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.643240Z" 2020-11-29T16:26:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "89" 2020-11-29T16:26:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.437925.78a31ae7", 2020-11-29T16:26:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.437925Z", 2020-11-29T16:26:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.134729Z" 2020-11-29T16:26:31.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.582 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.582 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.582 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.582 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.583 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.583 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.583 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "33" 2020-11-29T16:26:31.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.437992.7113d95b", 2020-11-29T16:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.437992Z", 2020-11-29T16:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.586 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.594569Z" 2020-11-29T16:26:31.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.586 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.586 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.587 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.587 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.587 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.587 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.588 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "66" 2020-11-29T16:26:31.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.589 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.438050.edb154b4", 2020-11-29T16:26:31.589 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.438050Z", 2020-11-29T16:26:31.589 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.590 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.590 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.342821Z" 2020-11-29T16:26:31.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.590 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.591 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.591 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.591 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.591 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.592 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.592 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "15" 2020-11-29T16:26:31.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.593 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.438109.74f6236f", 2020-11-29T16:26:31.593 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.593 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.438109Z", 2020-11-29T16:26:31.594 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.594 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.594 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.980248Z" 2020-11-29T16:26:31.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.595 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.595 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.595 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.595 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.596 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.596 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.596 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "52" 2020-11-29T16:26:31.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.597 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.438166.03c0916d", 2020-11-29T16:26:31.597 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.597 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.438166Z", 2020-11-29T16:26:31.598 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.598 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.598 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.254922Z" 2020-11-29T16:26:31.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.599 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.599 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.599 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.599 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.600 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.600 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.600 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "13" 2020-11-29T16:26:31.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.601 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.438222.b24f7158", 2020-11-29T16:26:31.601 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.601 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.438222Z", 2020-11-29T16:26:31.602 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.602 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.602 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.418634Z" 2020-11-29T16:26:31.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.603 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.603 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.603 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.603 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.604 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.604 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.604 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "62" 2020-11-29T16:26:31.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.605 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.438276.4a4f6a00", 2020-11-29T16:26:31.605 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.605 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.438276Z", 2020-11-29T16:26:31.606 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.606 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.606 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.826834Z" 2020-11-29T16:26:31.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.606 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.607 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.607 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.607 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.607 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.608 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.608 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "26" 2020-11-29T16:26:31.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.609 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.438335.0b794161", 2020-11-29T16:26:31.609 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.609 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.438335Z", 2020-11-29T16:26:31.610 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.610 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.610 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.622962Z" 2020-11-29T16:26:31.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.610 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.611 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.611 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.611 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.612 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.612 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.612 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "44" 2020-11-29T16:26:31.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.613 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.438860.0314858a", 2020-11-29T16:26:31.613 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.438860Z", 2020-11-29T16:26:31.614 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.614 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.614 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.180355Z" 2020-11-29T16:26:31.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.615 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.615 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.615 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.615 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.616 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.616 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.616 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "34" 2020-11-29T16:26:31.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.617 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667186.438919.247d3117", 2020-11-29T16:26:31.617 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.618 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:26.438919Z", 2020-11-29T16:26:31.618 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.618 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.618 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.467963Z" 2020-11-29T16:26:31.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.619 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.619 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.619 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.620 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.620 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.620 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.620 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "85" 2020-11-29T16:26:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.621 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.366112.49e3cff4", 2020-11-29T16:26:31.622 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.622 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.366112Z", 2020-11-29T16:26:31.622 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.622 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.623 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.650518Z" 2020-11-29T16:26:31.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.623 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.623 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.623 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.624 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.624 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.624 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.624 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "22" 2020-11-29T16:26:31.625 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.625 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.625 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.373416.40802d0f", 2020-11-29T16:26:31.626 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.626 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.373416Z", 2020-11-29T16:26:31.626 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.626 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.491095Z" 2020-11-29T16:26:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.627 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.628 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.628 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.628 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.628 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "41" 2020-11-29T16:26:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.629 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.373932.7c2f374c", 2020-11-29T16:26:31.630 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.630 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.373932Z", 2020-11-29T16:26:31.630 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.630 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.243028Z" 2020-11-29T16:26:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.631 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.632 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "58" 2020-11-29T16:26:31.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.634 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.380671.19325b9d", 2020-11-29T16:26:31.634 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.634 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.380671Z", 2020-11-29T16:26:31.634 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.506566Z" 2020-11-29T16:26:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.635 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.636 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "64" 2020-11-29T16:26:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.380848.d747a003", 2020-11-29T16:26:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.380848Z", 2020-11-29T16:26:31.638 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.935054Z" 2020-11-29T16:26:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.639 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.640 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.640 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.640 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.640 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.641 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "51" 2020-11-29T16:26:31.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.642 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.380929.70cf2196", 2020-11-29T16:26:31.642 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.380929Z", 2020-11-29T16:26:31.642 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.642 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.643 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.059065Z" 2020-11-29T16:26:31.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.643 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.643 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.643 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.644 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.644 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.644 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "76" 2020-11-29T16:26:31.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.645 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.380995.67e90fff", 2020-11-29T16:26:31.646 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.380995Z", 2020-11-29T16:26:31.646 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.646 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.647 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.358624Z" 2020-11-29T16:26:31.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.647 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.647 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.648 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.648 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.648 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.648 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.649 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "105" 2020-11-29T16:26:31.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.650 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.381219.4c5c10b5", 2020-11-29T16:26:31.650 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.650 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.381219Z", 2020-11-29T16:26:31.650 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.687291Z" 2020-11-29T16:26:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.651 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.652 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.652 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.652 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "90" 2020-11-29T16:26:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.654 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.385361.1c8d0ac0", 2020-11-29T16:26:31.654 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.655 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.385361Z", 2020-11-29T16:26:31.655 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.655 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.655 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.155064Z" 2020-11-29T16:26:31.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.656 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.656 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.656 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.656 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.657 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.657 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.657 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "56" 2020-11-29T16:26:31.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.658 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.385451.d139ff85", 2020-11-29T16:26:31.659 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.659 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.385451Z", 2020-11-29T16:26:31.659 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.659 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.659 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.971160Z" 2020-11-29T16:26:31.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.660 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.660 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.660 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.660 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "74" 2020-11-29T16:26:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.385515.25690d92", 2020-11-29T16:26:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.385515Z", 2020-11-29T16:26:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.014798Z" 2020-11-29T16:26:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "75" 2020-11-29T16:26:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.385579.31562880", 2020-11-29T16:26:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.385579Z", 2020-11-29T16:26:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.714624Z" 2020-11-29T16:26:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "46" 2020-11-29T16:26:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.670 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.670 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.386006.db0e3510", 2020-11-29T16:26:31.670 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.671 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.386006Z", 2020-11-29T16:26:31.671 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.671 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.671 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.518890Z" 2020-11-29T16:26:31.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.672 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.672 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.672 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.672 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.673 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.673 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.673 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "19" 2020-11-29T16:26:31.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.674 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.386084.2cb13c4b", 2020-11-29T16:26:31.674 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.674 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.386084Z", 2020-11-29T16:26:31.675 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.675 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.675 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.694937Z" 2020-11-29T16:26:31.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.675 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.676 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.676 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.676 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.677 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.677 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.677 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "23" 2020-11-29T16:26:31.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.681 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.386150.bfbfe9e9", 2020-11-29T16:26:31.681 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.682 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.386150Z", 2020-11-29T16:26:31.682 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.682 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.683 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.447286Z" 2020-11-29T16:26:31.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.683 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.683 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.683 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.684 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.684 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.684 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.684 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.684 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "107" 2020-11-29T16:26:31.685 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.685 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.685 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.386434.98f23def", 2020-11-29T16:26:31.686 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.686 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.386434Z", 2020-11-29T16:26:31.686 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.686 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.687 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.331837Z" 2020-11-29T16:26:31.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.687 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.687 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.687 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.688 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.688 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.688 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.688 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "60" 2020-11-29T16:26:31.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.689 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.386495.69b6feac", 2020-11-29T16:26:31.690 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.690 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.386495Z", 2020-11-29T16:26:31.690 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.690 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.690 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.871626Z" 2020-11-29T16:26:31.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.691 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.691 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.691 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.691 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.692 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.692 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.692 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "27" 2020-11-29T16:26:31.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.693 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.693 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.386534.d66f8cf1", 2020-11-29T16:26:31.693 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.694 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.386534Z", 2020-11-29T16:26:31.694 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.694 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.694 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.875156Z" 2020-11-29T16:26:31.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.695 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.695 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.695 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.695 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.695 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.696 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.696 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.696 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "72" 2020-11-29T16:26:31.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.387718.5ed557c5", 2020-11-29T16:26:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.697 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.387718Z", 2020-11-29T16:26:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.359490Z" 2020-11-29T16:26:31.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.699 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.699 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.699 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.699 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.700 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.700 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.700 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "38" 2020-11-29T16:26:31.700 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.700 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.701 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.387769.cc45f195", 2020-11-29T16:26:31.701 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.387769Z", 2020-11-29T16:26:31.702 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.702 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.702 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.267191Z" 2020-11-29T16:26:31.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.702 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.703 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.703 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.703 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.703 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.704 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.704 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "103" 2020-11-29T16:26:31.704 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.704 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.705 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.387851.fc6627b2", 2020-11-29T16:26:31.705 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.705 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.387851Z", 2020-11-29T16:26:31.705 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.706 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.706 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.311046Z" 2020-11-29T16:26:31.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.706 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.706 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.707 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.708 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "37" 2020-11-29T16:26:31.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.709 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.393127.f1490e5a", 2020-11-29T16:26:31.709 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.709 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.393127Z", 2020-11-29T16:26:31.709 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.710 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.710 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.819199Z" 2020-11-29T16:26:31.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.710 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.710 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.711 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.711 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.711 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.711 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.712 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "93" 2020-11-29T16:26:31.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.393475.3318cda0", 2020-11-29T16:26:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.393475Z", 2020-11-29T16:26:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.713 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.714 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.999661Z" 2020-11-29T16:26:31.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.714 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.714 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.715 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.715 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.715 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.715 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.716 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "97" 2020-11-29T16:26:31.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.717 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.394331.7b3a82f1", 2020-11-29T16:26:31.717 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.717 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.394331Z", 2020-11-29T16:26:31.717 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.717 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.718 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.335578Z" 2020-11-29T16:26:31.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.718 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.718 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.718 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.719 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.719 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.719 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.719 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "82" 2020-11-29T16:26:31.720 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.720 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.720 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.720 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.394501.f1bccf22", 2020-11-29T16:26:31.721 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.721 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.394501Z", 2020-11-29T16:26:31.721 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.721 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.721 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.194983Z" 2020-11-29T16:26:31.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.722 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.722 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.722 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.723 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.723 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.723 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "79" 2020-11-29T16:26:31.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.724 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.394859.5fea80a1", 2020-11-29T16:26:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.394859Z", 2020-11-29T16:26:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.491173Z" 2020-11-29T16:26:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.726 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.727 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.727 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.727 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "108" 2020-11-29T16:26:31.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.728 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.728 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.395004.7ffb94e2", 2020-11-29T16:26:31.728 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.729 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.395004Z", 2020-11-29T16:26:31.729 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.729 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.729 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.555928Z" 2020-11-29T16:26:31.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.730 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.730 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.730 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.730 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.731 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.731 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.731 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "87" 2020-11-29T16:26:31.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.732 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667187.528912.634b9094", 2020-11-29T16:26:31.732 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.733 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:27.528912Z", 2020-11-29T16:26:31.733 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.733 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.733 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.782916Z" 2020-11-29T16:26:31.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.734 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.734 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.734 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.734 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.735 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.735 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.735 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "25" 2020-11-29T16:26:31.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.736 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.428151.286bedcd", 2020-11-29T16:26:31.736 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.736 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.428151Z", 2020-11-29T16:26:31.737 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.737 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.737 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.130867Z" 2020-11-29T16:26:31.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.737 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.738 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.738 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.738 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.738 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.739 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.739 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "100" 2020-11-29T16:26:31.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.740 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.428377.089fded0", 2020-11-29T16:26:31.740 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.740 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.428377Z", 2020-11-29T16:26:31.740 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.741 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.741 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.759197Z" 2020-11-29T16:26:31.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.741 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.741 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.742 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.742 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.742 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.743 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "47" 2020-11-29T16:26:31.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.744 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.428521.adadb4b4", 2020-11-29T16:26:31.744 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.428521Z", 2020-11-29T16:26:31.744 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.744 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.745 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.430987Z" 2020-11-29T16:26:31.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.745 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.745 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.745 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.746 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.746 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.747 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "17" 2020-11-29T16:26:31.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.748 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.428690.8131d5b7", 2020-11-29T16:26:31.748 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.428690Z", 2020-11-29T16:26:31.748 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.748 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.749 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.867248Z" 2020-11-29T16:26:31.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.749 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.749 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.749 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.750 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.750 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.750 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.750 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "94" 2020-11-29T16:26:31.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.751 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.429910.705f8473", 2020-11-29T16:26:31.752 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.752 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.429910Z", 2020-11-29T16:26:31.752 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.752 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.752 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.424119Z" 2020-11-29T16:26:31.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.753 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.753 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.753 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.753 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.754 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.754 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.754 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "84" 2020-11-29T16:26:31.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.755 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.430725.1ee1ba76", 2020-11-29T16:26:31.755 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.756 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.430725Z", 2020-11-29T16:26:31.756 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.756 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.756 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.551072Z" 2020-11-29T16:26:31.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.757 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.757 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.757 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.757 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.758 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.758 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.758 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "65" 2020-11-29T16:26:31.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.759 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.430832.710fb2f6", 2020-11-29T16:26:31.759 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.760 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.430832Z", 2020-11-29T16:26:31.760 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.760 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.760 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.003096Z" 2020-11-29T16:26:31.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.761 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.761 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.761 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.761 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.762 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.762 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.762 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "30" 2020-11-29T16:26:31.762 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.762 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.763 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.431206.b4e1a9f6", 2020-11-29T16:26:31.763 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.763 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.431206Z", 2020-11-29T16:26:31.764 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.764 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.764 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.298777Z" 2020-11-29T16:26:31.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.764 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.765 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.765 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.765 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.765 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.766 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.766 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "14" 2020-11-29T16:26:31.766 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.766 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.767 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.432130.29680b11", 2020-11-29T16:26:31.767 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.767 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.432130Z", 2020-11-29T16:26:31.768 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.768 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.768 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.578846Z" 2020-11-29T16:26:31.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.768 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.769 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.769 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.769 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.769 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.770 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.770 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "43" 2020-11-29T16:26:31.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.771 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.432286.ba402fce", 2020-11-29T16:26:31.771 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.771 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.432286Z", 2020-11-29T16:26:31.771 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.772 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.772 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.087306Z" 2020-11-29T16:26:31.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.772 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.772 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.773 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.773 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.773 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.773 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.774 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "99" 2020-11-29T16:26:31.774 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.774 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.775 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.432383.dd69fe49", 2020-11-29T16:26:31.775 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.775 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.432383Z", 2020-11-29T16:26:31.775 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.776 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.776 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.562633Z" 2020-11-29T16:26:31.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.776 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.776 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.777 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.777 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.777 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.777 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.778 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "20" 2020-11-29T16:26:31.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.779 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.432483.ae4a169d", 2020-11-29T16:26:31.779 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.779 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.432483Z", 2020-11-29T16:26:31.779 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.779 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.780 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.534940Z" 2020-11-29T16:26:31.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.780 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.780 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.780 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.781 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.781 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.781 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.781 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "109" 2020-11-29T16:26:31.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:31.783 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.432672.d351de38", 2020-11-29T16:26:31.783 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:31.784 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.432672Z", 2020-11-29T16:26:31.784 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:31.784 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.827147Z" 2020-11-29T16:26:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "71" 2020-11-29T16:26:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:31.788 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:31.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'kcmyep-82_sub', '--marker', 'kcmyep-82_sub/1606667190.432672.d351de38', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:31.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'subscription', 'pull', '--uid', 'rgw-multisite-test-user', '--subscription', 'kcmyep-82_sub', '--marker', 'kcmyep-82_sub/1606667190.432672.d351de38', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-11-29T16:26:31.789 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 subscription pull --uid rgw-multisite-test-user --subscription kcmyep-82_sub --marker kcmyep-82_sub/1606667190.432672.d351de38 --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-11-29T16:26:31.871 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:31.872 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:32.106 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "next_marker": "", 2020-11-29T16:26:32.106 INFO:teuthology.orchestra.run.smithi184.stdout: "is_truncated": "false", 2020-11-29T16:26:32.107 INFO:teuthology.orchestra.run.smithi184.stdout: "events": [ 2020-11-29T16:26:32.107 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.107 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.437050.2a0b50d8", 2020-11-29T16:26:32.107 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.108 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.437050Z", 2020-11-29T16:26:32.108 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.108 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.108 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.379107Z" 2020-11-29T16:26:32.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.109 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.109 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.109 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.109 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.110 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.110 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.110 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "83" 2020-11-29T16:26:32.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.111 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.111 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.111 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.437348.99e58070", 2020-11-29T16:26:32.111 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.112 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.437348Z", 2020-11-29T16:26:32.112 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.112 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.112 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.958406Z" 2020-11-29T16:26:32.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.113 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.113 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.113 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.114 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.114 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.114 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.114 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "29" 2020-11-29T16:26:32.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.115 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.116 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.437457.a298ee85", 2020-11-29T16:26:32.116 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.116 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.437457Z", 2020-11-29T16:26:32.116 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.116 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.117 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.211247Z" 2020-11-29T16:26:32.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.117 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.117 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.117 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.118 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.118 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.118 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.118 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "12" 2020-11-29T16:26:32.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.119 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.437561.453cb387", 2020-11-29T16:26:32.120 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.120 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.437561Z", 2020-11-29T16:26:32.120 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.120 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.120 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.222788Z" 2020-11-29T16:26:32.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.121 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.121 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.121 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.121 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.122 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.122 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.122 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "35" 2020-11-29T16:26:32.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.123 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.437963.e3e5bc15", 2020-11-29T16:26:32.124 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.124 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.437963Z", 2020-11-29T16:26:32.124 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.124 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.124 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.022959Z" 2020-11-29T16:26:32.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.125 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.125 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.125 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.125 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "53" 2020-11-29T16:26:32.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.438275.1384f2fe", 2020-11-29T16:26:32.127 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.438275Z", 2020-11-29T16:26:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:16.386206Z" 2020-11-29T16:26:32.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "16" 2020-11-29T16:26:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.438391.e0a73eea", 2020-11-29T16:26:32.131 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.132 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.438391Z", 2020-11-29T16:26:32.132 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.132 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.132 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:20.044044Z" 2020-11-29T16:26:32.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.133 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.135 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.136 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.136 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.136 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.137 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.137 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "98" 2020-11-29T16:26:32.137 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.137 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.138 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.438750.94e50cdf", 2020-11-29T16:26:32.138 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.138 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.438750Z", 2020-11-29T16:26:32.139 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.139 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.139 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:19.511508Z" 2020-11-29T16:26:32.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.139 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.140 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.140 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.140 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.140 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.141 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.141 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "86" 2020-11-29T16:26:32.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.142 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.142 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.438870.de945e22", 2020-11-29T16:26:32.142 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.142 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.438870Z", 2020-11-29T16:26:32.142 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.143 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.143 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.682866Z" 2020-11-29T16:26:32.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.143 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.144 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.144 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.144 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.144 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.145 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.145 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "68" 2020-11-29T16:26:32.145 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.145 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.146 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.146 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.438968.167da1ff", 2020-11-29T16:26:32.146 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.146 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.438968Z", 2020-11-29T16:26:32.146 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.147 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.147 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:18.374954Z" 2020-11-29T16:26:32.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.147 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.148 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.148 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.148 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.148 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.149 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.149 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "61" 2020-11-29T16:26:32.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.150 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.150 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.439190.ff29437c", 2020-11-29T16:26:32.150 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.150 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.439190Z", 2020-11-29T16:26:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.670679Z" 2020-11-29T16:26:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.151 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.152 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.152 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.152 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.152 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.153 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.153 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "45" 2020-11-29T16:26:32.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.154 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.154 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1606667190.439478.8803b808", 2020-11-29T16:26:32.154 INFO:teuthology.orchestra.run.smithi184.stdout: "event": "OBJECT_CREATE", 2020-11-29T16:26:32.154 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:30.439478Z", 2020-11-29T16:26:32.154 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.155 INFO:teuthology.orchestra.run.smithi184.stdout: "attrs": { 2020-11-29T16:26:32.155 INFO:teuthology.orchestra.run.smithi184.stdout: "mtime": "2020-11-29T16:26:17.046934Z" 2020-11-29T16:26:32.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.155 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket": { 2020-11-29T16:26:32.155 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_id": "a1734b2a-fbe5-446c-a86c-68fcf5b64a76.5484.30", 2020-11-29T16:26:32.156 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "kcmyep-82", 2020-11-29T16:26:32.156 INFO:teuthology.orchestra.run.smithi184.stdout: "tenant": "" 2020-11-29T16:26:32.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.156 INFO:teuthology.orchestra.run.smithi184.stdout: "key": { 2020-11-29T16:26:32.156 INFO:teuthology.orchestra.run.smithi184.stdout: "instance": "", 2020-11-29T16:26:32.157 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "31" 2020-11-29T16:26:32.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.158 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:32.158 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:32.159 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2020-11-29T16:26:32.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:32.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:32.161 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:32.204 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:32.205 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:32.449 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:26:32.488 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:26:32.488 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:32.488 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:32.488 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:32.489 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:32.489 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:32.489 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667174.684079_542.1", 2020-11-29T16:26:32.489 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:26:14.684079Z" 2020-11-29T16:26:32.489 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:32.490 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:32.490 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:32.490 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:32.490 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:32.491 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:32.491 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:32.491 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:32.491 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:32.491 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:26:32.492 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:26:32.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:32.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:32.494 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:32.535 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:32.536 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:32.816 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:32.817 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:26:32.817 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:32.817 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:26:32.818 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:26:32.818 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:32.818 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:32.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.819 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:26:32.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.819 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:26:32.820 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:32.820 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:32.820 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:32.820 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:32.820 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:26:32.821 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:32.821 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:32.822 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:32.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.822 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:26:32.823 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:32.823 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:32.823 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667174.684079_542.1", 2020-11-29T16:26:32.823 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:32.824 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:32.824 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:32.824 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:14.684079Z", 2020-11-29T16:26:32.825 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:32.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.825 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.825 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:26:32.826 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:32.826 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:32.826 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:32.826 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:32.827 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:32.827 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:32.827 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:32.827 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:32.827 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.828 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:32.828 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:26:32.828 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:32.829 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:32.829 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:32.829 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:32.829 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:32.829 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:32.830 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:32.830 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:32.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.831 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.831 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:32.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:32.831 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:26:32.832 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:26:32.832 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:26:32.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:32.832 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:32.834 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:26:32.834 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:26:32.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:32.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:32.836 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:32.877 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:32.878 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:33.154 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:26:33.154 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:26:33.154 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:26:33.154 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:26:33.155 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:26:33.155 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:33.155 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:26:33.155 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:33.156 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:26:33.156 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:33.156 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:26:33.156 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:33.156 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:33.157 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:33.157 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:33.157 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:26:33.157 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:33.157 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:33.158 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:33.158 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:33.158 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:33.158 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:33.159 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:26:33.159 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:33.159 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:33.159 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667174.684079_542.1", 2020-11-29T16:26:33.159 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:33.160 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:33.160 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:33.160 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:26:14.684079Z", 2020-11-29T16:26:33.160 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:26:33.160 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:33.161 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:33.161 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:33.161 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:26:33.161 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:33.161 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:33.162 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:33.162 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:33.162 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:33.163 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:33.163 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:26:33.163 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:33.163 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:33.164 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:33.164 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:33.164 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:26:33.164 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:33.164 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:33.165 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:33.165 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:33.165 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:33.165 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:33.166 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:26:33.166 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:33.166 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:33.166 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:33.166 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:26:33.167 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:33.167 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:26:33.167 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:26:33.167 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:26:33.167 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:33.168 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:26:33.169 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:26:33.169 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:26:33.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:33.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:33.171 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:33.213 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:33.213 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:33.584 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:33.584 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:26:33.585 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:33.585 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:26:33.585 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:26:33.586 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:33.586 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:33.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:33.586 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:26:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:26:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:33.587 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:33.588 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:33.588 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:33.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:26:33.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:33.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:33.590 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:33.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:33.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:33.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:33.591 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:26:33.591 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:33.591 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:33.591 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667174.684079_542.1", 2020-11-29T16:26:33.592 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:33.592 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:33.592 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:33.592 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:14.684079Z", 2020-11-29T16:26:33.593 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:33.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:33.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:33.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:33.594 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:26:33.594 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:33.595 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:33.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:33.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:26:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:33.598 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:33.599 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:33.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:33.601 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:33.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:33.601 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:26:33.601 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:26:33.602 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:26:33.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:33.602 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:33.603 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:26:33.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:33.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:33.606 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:33.647 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:33.648 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:33.890 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:26:33.925 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:26:33.925 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:33.925 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:33.925 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:33.926 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:33.926 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:33.926 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667174.684079_542.1", 2020-11-29T16:26:33.926 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:26:14.684079Z" 2020-11-29T16:26:33.927 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:33.927 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:33.927 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:33.927 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:33.927 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:33.928 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:33.928 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:33.928 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:33.928 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:33.928 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:26:33.929 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:26:33.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:33.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:33.931 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:33.971 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:33.971 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:34.341 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:34.341 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:26:34.342 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:34.342 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:26:34.342 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:26:34.343 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:34.343 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:34.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:34.344 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:26:34.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:34.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:26:34.345 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:34.346 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:34.346 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:34.347 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:34.347 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:26:34.347 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:34.348 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:34.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:34.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:34.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:34.349 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:34.350 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:26:34.350 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:34.351 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:34.351 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667174.684079_542.1", 2020-11-29T16:26:34.351 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:34.352 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:34.353 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:14.684079Z", 2020-11-29T16:26:34.353 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:34.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:34.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:34.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:34.355 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:26:34.355 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:34.356 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:34.356 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:34.357 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:34.357 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:34.357 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:34.358 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:34.358 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:34.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:26:34.360 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:26:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:26:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:26:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:34.367 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:34.368 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-11-29T16:26:34.559 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-11-29T16:26:34.550+0000 7f9a8a7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:26:34.560 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-11-29T16:26:34.554+0000 7fc9c2ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:26:34.579 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:26:34.573+0000 7fec0affd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:26:34.580 INFO:tasks.rgw.c1.client.1.smithi168.stdout:2020-11-29T16:26:34.573+0000 7fec0b7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-11-29T16:26:34.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:26:34.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:26:34.888 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:26:34.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:34.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:35.173 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:35.166+0000 7f0262267080 1 bucket sync caught up with source: 2020-11-29T16:26:35.174 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.129.5, , 00000000001.94.5, 00000000001.191.5, 00000000001.88.5, , 00000000001.139.5, 00000000001.72.5, 00000000001.116.5, 00000000002.84.5] 2020-11-29T16:26:35.174 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.129.5, , 00000000001.94.5, 00000000001.191.5, 00000000001.88.5, , 00000000001.139.5, 00000000001.72.5, 00000000001.116.5, 00000000002.84.5] 2020-11-29T16:26:35.175 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:35.166+0000 7f0262267080 0 bucket checkpoint complete 2020-11-29T16:26:35.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:26:35.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'kcmyep-83', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-11-29T16:26:35.760 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket kcmyep-83 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-11-29T16:26:35.809 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:35.810 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:36.058 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:36.050+0000 7f4c7e727080 1 waiting for incremental sync to catch up: 2020-11-29T16:26:36.059 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.45.5, 00000000002.129.5, , 00000000001.94.5, 00000000001.191.5, 00000000001.88.5, , 00000000001.139.5, 00000000001.72.5, 00000000001.116.5, 00000000002.84.5] 2020-11-29T16:26:36.059 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.45.5, 00000000003.131.5, , 00000000002.96.5, 00000000001.191.5, 00000000003.92.5, , 00000000003.143.5, 00000000002.74.5, 00000000002.118.5, 00000000003.86.5] 2020-11-29T16:26:41.063 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:41.058+0000 7f4c7e727080 1 waiting for incremental sync to catch up: 2020-11-29T16:26:41.063 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.45.5, 00000000002.129.5, , 00000000001.94.5, 00000000001.191.5, 00000000001.88.5, , 00000000001.139.5, 00000000001.72.5, 00000000001.116.5, 00000000002.84.5] 2020-11-29T16:26:41.064 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.45.5, 00000000003.131.5, , 00000000002.96.5, 00000000001.191.5, 00000000003.92.5, , 00000000003.143.5, 00000000002.74.5, 00000000002.118.5, 00000000003.86.5] 2020-11-29T16:26:46.068 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:46.062+0000 7f4c7e727080 1 waiting for incremental sync to catch up: 2020-11-29T16:26:46.068 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.45.5, 00000000002.129.5, , 00000000001.94.5, 00000000001.191.5, 00000000001.88.5, , 00000000001.139.5, 00000000001.72.5, 00000000001.116.5, 00000000002.84.5] 2020-11-29T16:26:46.069 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.45.5, 00000000003.131.5, , 00000000002.96.5, 00000000001.191.5, 00000000003.92.5, , 00000000003.143.5, 00000000002.74.5, 00000000002.118.5, 00000000003.86.5] 2020-11-29T16:26:51.073 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:51.065+0000 7f4c7e727080 1 bucket sync caught up with source: 2020-11-29T16:26:51.074 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [00000000001.45.5, 00000000003.131.5, , 00000000002.96.5, 00000000001.191.5, 00000000003.92.5, , 00000000003.143.5, 00000000002.74.5, 00000000002.118.5, 00000000003.86.5] 2020-11-29T16:26:51.074 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [00000000001.45.5, 00000000003.131.5, , 00000000002.96.5, 00000000001.191.5, 00000000003.92.5, , 00000000003.143.5, 00000000002.74.5, 00000000002.118.5, 00000000003.86.5] 2020-11-29T16:26:51.075 INFO:teuthology.orchestra.run.smithi184.stderr:2020-11-29T16:26:51.065+0000 7f4c7e727080 0 bucket checkpoint complete 2020-11-29T16:26:51.299 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "10" type: "OBJECT_CREATE" 2020-11-29T16:26:51.300 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "18" type: "OBJECT_CREATE" 2020-11-29T16:26:51.300 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "13" type: "OBJECT_CREATE" 2020-11-29T16:26:51.301 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "15" type: "OBJECT_CREATE" 2020-11-29T16:26:51.301 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "11" type: "OBJECT_CREATE" 2020-11-29T16:26:51.301 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "19" type: "OBJECT_CREATE" 2020-11-29T16:26:51.302 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "14" type: "OBJECT_CREATE" 2020-11-29T16:26:51.302 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "12" type: "OBJECT_CREATE" 2020-11-29T16:26:51.303 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "16" type: "OBJECT_CREATE" 2020-11-29T16:26:51.303 DEBUG:tasks.rgw_multi.tests_ps:Event: objname: "17" type: "OBJECT_CREATE" 2020-11-29T16:26:51.624 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... ok 2020-11-29T16:26:51.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:51.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:51.626 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:51.662 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:51.663 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:51.901 INFO:teuthology.orchestra.run.smithi168.stderr:No --period given, using current period=24836445-da90-4fa5-864b-eda958e6bd14 2020-11-29T16:26:51.931 INFO:teuthology.orchestra.run.smithi168.stdout:[ 2020-11-29T16:26:51.931 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:51.931 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:51.932 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:51.932 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:51.932 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:51.932 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667211.616043_548.1", 2020-11-29T16:26:51.932 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "2020-11-29T16:26:51.616043Z" 2020-11-29T16:26:51.933 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:51.933 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:51.933 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:51.933 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:51.934 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:51.934 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:51.934 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:51.934 INFO:teuthology.orchestra.run.smithi168.stdout: "last_update": "0.000000" 2020-11-29T16:26:51.934 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:51.935 INFO:teuthology.orchestra.run.smithi168.stdout:] 2020-11-29T16:26:51.936 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-11-29T16:26:51.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:51.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:51.937 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:51.979 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:51.979 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:52.327 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-11-29T16:26:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-11-29T16:26:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-11-29T16:26:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-11-29T16:26:52.328 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-11-29T16:26:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:52.329 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:52.330 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-11-29T16:26:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-11-29T16:26:52.331 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:52.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-11-29T16:26:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:52.333 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:52.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-11-29T16:26:52.335 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:52.336 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:52.336 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1606667211.616043_548.1", 2020-11-29T16:26:52.336 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:52.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:52.337 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:52.337 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-11-29T16:26:51.616043Z", 2020-11-29T16:26:52.338 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-11-29T16:26:52.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:52.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-11-29T16:26:52.339 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:52.340 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:52.340 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:52.341 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:52.341 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:52.341 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:52.342 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:52.342 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:52.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:52.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:52.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-11-29T16:26:52.343 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-11-29T16:26:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-11-29T16:26:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-11-29T16:26:52.344 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-11-29T16:26:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-11-29T16:26:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-11-29T16:26:52.345 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-11-29T16:26:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-11-29T16:26:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-11-29T16:26:52.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-11-29T16:26:52.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-11-29T16:26:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-11-29T16:26:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-11-29T16:26:52.348 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-11-29T16:26:52.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-11-29T16:26:52.349 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-11-29T16:26:52.351 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-11-29T16:26:52.351 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-11-29T16:26:52.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:52.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:52.353 INFO:teuthology.orchestra.run.smithi168:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:52.396 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:52.397 INFO:teuthology.orchestra.run.smithi168.stderr:ignoring --setgroup ceph since I am not root 2020-11-29T16:26:52.657 INFO:teuthology.orchestra.run.smithi168.stdout:{ 2020-11-29T16:26:52.657 INFO:teuthology.orchestra.run.smithi168.stdout: "sync_status": { 2020-11-29T16:26:52.658 INFO:teuthology.orchestra.run.smithi168.stdout: "info": { 2020-11-29T16:26:52.658 INFO:teuthology.orchestra.run.smithi168.stdout: "status": "sync", 2020-11-29T16:26:52.658 INFO:teuthology.orchestra.run.smithi168.stdout: "num_shards": 4, 2020-11-29T16:26:52.658 INFO:teuthology.orchestra.run.smithi168.stdout: "period": "24836445-da90-4fa5-864b-eda958e6bd14", 2020-11-29T16:26:52.659 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:26:52.659 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:52.659 INFO:teuthology.orchestra.run.smithi168.stdout: "markers": [ 2020-11-29T16:26:52.659 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:52.659 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 0, 2020-11-29T16:26:52.660 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:52.660 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:52.660 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606665599.895400_96.1", 2020-11-29T16:26:52.661 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:52.661 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 1, 2020-11-29T16:26:52.661 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:52.661 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T15:59:59.895400Z", 2020-11-29T16:26:52.662 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:52.662 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:52.662 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:52.662 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:52.663 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 1, 2020-11-29T16:26:52.663 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:52.663 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:52.663 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "1_1606667211.616043_548.1", 2020-11-29T16:26:52.663 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:52.664 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:52.664 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:52.664 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "2020-11-29T16:26:51.616043Z", 2020-11-29T16:26:52.664 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 4 2020-11-29T16:26:52.665 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:52.665 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:52.665 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:52.666 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 2, 2020-11-29T16:26:52.666 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:52.666 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:52.667 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:52.667 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:52.667 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:52.668 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:52.668 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:26:52.668 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:52.668 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:52.669 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:52.669 INFO:teuthology.orchestra.run.smithi168.stdout: { 2020-11-29T16:26:52.669 INFO:teuthology.orchestra.run.smithi168.stdout: "key": 3, 2020-11-29T16:26:52.670 INFO:teuthology.orchestra.run.smithi168.stdout: "val": { 2020-11-29T16:26:52.670 INFO:teuthology.orchestra.run.smithi168.stdout: "state": 1, 2020-11-29T16:26:52.670 INFO:teuthology.orchestra.run.smithi168.stdout: "marker": "", 2020-11-29T16:26:52.670 INFO:teuthology.orchestra.run.smithi168.stdout: "next_step_marker": "", 2020-11-29T16:26:52.670 INFO:teuthology.orchestra.run.smithi168.stdout: "total_entries": 0, 2020-11-29T16:26:52.671 INFO:teuthology.orchestra.run.smithi168.stdout: "pos": 0, 2020-11-29T16:26:52.671 INFO:teuthology.orchestra.run.smithi168.stdout: "timestamp": "0.000000", 2020-11-29T16:26:52.671 INFO:teuthology.orchestra.run.smithi168.stdout: "realm_epoch": 2 2020-11-29T16:26:52.671 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:52.672 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:52.672 INFO:teuthology.orchestra.run.smithi168.stdout: ] 2020-11-29T16:26:52.672 INFO:teuthology.orchestra.run.smithi168.stdout: }, 2020-11-29T16:26:52.673 INFO:teuthology.orchestra.run.smithi168.stdout: "full_sync": { 2020-11-29T16:26:52.673 INFO:teuthology.orchestra.run.smithi168.stdout: "total": 1, 2020-11-29T16:26:52.673 INFO:teuthology.orchestra.run.smithi168.stdout: "complete": 1 2020-11-29T16:26:52.673 INFO:teuthology.orchestra.run.smithi168.stdout: } 2020-11-29T16:26:52.673 INFO:teuthology.orchestra.run.smithi168.stdout:} 2020-11-29T16:26:52.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-11-29T16:26:52.675 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-11-29T16:26:52.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:52.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-11-29T16:26:52.677 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-11-29T16:26:52.719 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-11-29T16:26:52.719 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-11-30T03:44:29.053 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-11-30T03:44:29.090 DEBUG:teuthology.task.console_log:Killing console logger for smithi168 2020-11-30T03:44:29.091 DEBUG:teuthology.task.console_log:Killing console logger for smithi184 2020-11-30T03:44:29.092 DEBUG:teuthology.task.console_log:Killing console logger for smithi168 2020-11-30T03:44:29.092 DEBUG:teuthology.task.console_log:Killing console logger for smithi184 2020-11-30T03:44:29.092 DEBUG:teuthology.exit:Finished running handlers